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Abstract 
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topics that are of interest to Python programmers, and (3) a Python workbook with 
lots of exercises. 
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Preface 


This book is a collection of materials that I've used when conducting Python training and 
also materials from my Web site that are intended for self-instruction. 


You may prefer a machine readable copy of this book. You can find it in various formats 
here: 


e HTML - http://www.davekuhlman.org/python_book_01.html 

e PDF -- http://www.davekuhlman.org /python_book_01.pdf 

e ODF/OpenOffice -- http://www.davekuhlman.org /python_book_01.odt 
And, let me thank the students in my Python classes. Their questions and suggestions 
were a great help in the preparation of these materials. 
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1 Part 1 -- Beginning Python 


1.1 Introductions Etc 

Introductions 

Practical matters: restrooms, breakroom, lunch and break times, etc. 
Starting the Python interactive interpreter. Also, [Python and Idle. 
Running scripts 


Editors -- Choose an editor which you can configure so that it indents with 4 spaces, not 
tab characters. For a list of editors for Python, see: 
http://wiki.python.org/moin/PythonEditors. A few possible editors: 


e SciTE -- http://www.scintilla.org/SciTE.html. 
e MS Windows only -- (1) TextPad -- http://www.textpad.com; (2) UltraEdit -- 
http://www.ultraedit.com/. 
e Jed -- See http://www.jedsoft.org/jed/. 
e Emacs -- See http://www. gnu.org/software/emacs/ and 
http://www.xemacs.org/faq/xemacs-faq.html. 
jEdit -- Requires a bit of customization for Python -- See http://jedit.org. 
Vim -- http://www.vim.org/ 
Geany -- http://www.geany.org/ 
e And many more. 
Interactive interpreters: 


e python 
e ipython 
e Idle 
IDEs -- Also see 
http://en. wikipedia.org/wiki/List_of_integrated_development_environments_for_Python: 


e PyWin -- MS Windows only. Available at: 
http://sourceforge.net/projects/pywin32/. 

WingIDE -- See http://wingware.com/wingide/. 

Eclipse -- http://eclipse.org/. There is a plug-in that supports Python. 
Kdevelop -- Linux/KDE -- See http://www.kdevelop.org/. 

Eric -- Linux KDE? -- See http://eric-ide.python-projects.org/index.html 
Emacs and SciTE will evaluate a Python buffer within the editor. 
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1.1.1 Resources 
Where else to get help: 


e Python home page -- http://www.python.org 

e Python standard documentation -- http://www.python.org/doc/. 
You will also find links to tutorials there. 

e FAQs -- http://www.python.org/doc/faq/. 

e The Python Wiki -- http://wiki.python.org/ 

e The Python Package Index -- Lots of Python packages -- 
https://pypi.python.org/pypi 

e Special interest groups (SIGs) -- http://www.python.org/sigs/ 

e Other python related mailing lists and lists for specific applications (for example, 
Zope, Twisted, etc). Try: http://dir.gmane.org/search.php?match=python. 

e http://sourceforge.net -- Lots of projects. Search for "python". 

e USENET -- comp.lang.python. Can also be accessed through Gmane: 
http://dir.gmane.org/gmane.comp.python. general. 

e The Python tutor email list -- http://mail.python.org/mailman/listinfo/tutor 

Local documentation: 


e OnMS Windows, the Python documentation is installed with the standard 
installation. 

e Install the standard Python documentation on your machine from 
http://www.python.org/doc/. 

e pydoc. Example, on the command line, type: pydoc re. 


e Import a module, then view its .__doc__ attribute. 
e At the interactive prompt, use help (obj). You might need to import it first. 
Example: 


2S almeeie hellAilalle) 
Se lave lijes (bie Il Walioy)) 


e InIPython, the question mark operator gives help. Example: 


in) Siig tepen? 


Type: builtin_function_or_method 
Base Class: <type 'builtin_function_or_method'> 
SENG] Iionaiae —DuLle—in function open 
Namespace: Python buailein 
Dosing mmc 
open(name[, mode[, buffering]]) -> file object 


Open a file using the file() type, returns a file 


object. 
COMISiicbiCieOie Wie Scie liaye(s 

Oye ee inal el Mee epee) clglaln cules illahb aes sce isyeie: 
og le elsis) a ioleye! for signature 
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Cabana Yes 
Call def: Calling definition not available.Call 
docstring: 

$5 Calli, Secs.) <== > Soe) 


1.1.2 A general description of Python 


Python is a high-level general purpose programming language: 


Because code is automatically compiled to byte code and executed, Python is 
suitable for use as a scripting language, Web application implementation 
language, etc. 

Because Python can be extended in C and C++, Python can provide the speed 
needed for even compute intensive tasks. 

Because of its strong structuring constructs (nested code blocks, functions, 
classes, modules, and packages) and its consistent use of objects and 
object-oriented programming, Python enables us to write clear, logical 
applications for small and large tasks. 


Important features of Python: 


Built-in high level data types: strings, lists, dictionaries, etc. 

The usual control structures: if, if-else, if-elif-else, while, plus a powerful 
collection iterator (for). 

Multiple levels of organizational structure: functions, classes, modules, and 
packages. These assist in organizing code. An excellent and large example is the 
Python standard library. 

Compile on the fly to byte code -- Source code is compiled to byte code without a 
separate compile step. Source code modules can also be "pre-compiled" to byte 
code files. 

Object-oriented -- Python provides a consistent way to use objects: everything is 
an object. And, in Python it is easy to implement new object types (called classes 
in object-oriented programming). 

Extensions in C and C++ -- Extension modules and extension types can be written 
by hand. There are also tools that help with this, for example, SWIG, sip, Pyrex. 
Jython is a version of Python that "plays well with" Java. See: The Jython Project 
-- http://www.jython.org/Project/. 


Some things you will need to know: 


Python uses indentation to show block structure. Indent one level to show the 
beginning of a block. Out-dent one level to show the end of a block. As an 
example, the following C-style code: 


alse {(@<)) 
{ 
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£2 () 


And, the convention is to use four spaces (and no hard tabs) for each level of indentation. 
Actually, it's more than a convention; it's practically a requirement. Following that 
"convention" will make it so much easier to merge your Python code with code from 
other sources. 


An overview of Python: 


e A scripting language -- Python is suitable (1) for embedding, (2) for writing small 
unstructured scripts, (3) for "quick and dirty" programs. 

e Not ascripting language -- (1) Python scales. (2) Python encourages us to write 
code that is clear and well-structured. 

e Interpreted, but also compiled to byte-code. Modules are automatically compiled 
(to .pyc) when imported, but may also be explicitly compiled. 

e Provides an interactive command line and interpreter shell. In fact, there are 
several. 

e Dynamic -- For example: 

Types are bound to values, not to variables. 

Function and method lookup is done at runtime. 

Values are inspect-able. 

There is an interactive interpreter, more than one, in fact. 

You can list the methods supported by any given object. 

e Strongly typed at run-time, not compile-time. Objects (values) have a type, but 
variables do not. 

e Reasonably high level -- High level built-in data types; high level control 
structures (for walking lists and iterators, for example). 

e Object-oriented -- Almost everything is an object. Simple object definition. Data 
hiding by agreement. Multiple inheritance. Interfaces by convention. 
Polymorphism. 

e Highly structured -- Statements, functions, classes, modules, and packages enable 
us to write large, well-structured applications. Why structure? Readability, 
locate-ability, modifiability. 

e Explicitness 


OF OOF Or “0 
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First-class objects: 

o Definition: Can (1) pass to function; (2) return from function; (3) stuff into a 
data structure. 

o Operators can be applied to values (not variables). Example: f (x) [3] 

Indented block structure -- "Python is pseudo-code that runs." 

Embedding and extending Python -- Python provides a well-documented and 

supported way (1) to embed the Python interpreter in C/C++ applications and (2) 

to extend Python with modules and objects implemented in C/C++. 

o Insome cases, SWIG can generate wrappers for existing C/C++ code 
automatically. See http://www.swig.org/ 

o Cython enables us to generate C code from Python and to "easily" create 
wrappers for C/C++ functions. See 
http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/ 

o To embed and extend Python with Java, there is Jython. See 
http://www.jython.org/ 

Automatic garbage collection. (But, there is a gc module to allow explicit control 

of garbage collection.) 

Comparison with other languages: compiled languages (e.g. C/C++); Java; Perl, 

Tcl, and Ruby. Python excells at: development speed, execution speed, clarity and 

maintainability. 

Varieties of Python: 

o CPython -- Standard Python 2.x implemented in C. 

o Jython -- Python for the Java environment -- http://www.jython.org/ 

o PyPy -- Python with a JIT compiler and stackless mode -- http://pypy.org/ 

o Stackless -- Python with enhanced thread support and microthreads etc. -- 

http://www.stackless.com/ 

IronPython -- Python for .NET and the CLR -- http://ironpython.net/ 

o Python 3 -- The new, new Python. This is intended as a replacement for 
Python 2.x. -- http://www.python.org/doc/. A few differences (from Python 
2.x): 

m The print statement changed to the print function. 

Strings are unicode by default. 

Classes are all "new style" classes. 

Changes to syntax for catching exceptions. 

Changes to integers -- no long integer; integer division with automatic 

convert to float. 

= More pervasive use of iterables (rather than collections). 

= Etc. 

For a more information about differences between Python 2.x and Python 3.x, 

see the description of the various fixes that can be applied with the 2t03 tool: 


e) 
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http://docs.python.org/3/library/2to3 .html#fixers 
The migration tool, 2t.03, eases the conversion of 2.x code to 3.x. 
e Also see The Zen of Python -- http://www.python.org/peps/pep-0020.html. Or, at 
the Python interactive prompt, type: 


Poe Impore, thas 


1.1.3 Interactive Python 


If you execute Python from the command line with no script (no arguments), Python 
gives you an interactive prompt. This is an excellent facility for learning Python and for 
trying small snippets of code. Many of the examples that follow were developed using 
the Python interactive prompt. 


Start the Python interactive interpreter by typing python with no arguments at the 
command line. For example: 


S python 

Pyenonm Zions AeA Ieson Silko, wlan Ii 2 OOS oe 19) 23) 

[GCC 4.3.2] on linux2 

Type "help", "copyright", "credits" or "license" for more 
information. 

SSS jonetinng aveilike 

hello 

>>> 


You may also want to consider using IDLE. IDLE is a graphical integrated development 
environment for Python; it contains a Python shell. It is likely that Idle was installed for 
you when you installed Python. You will find a script to start up IDLE in the 
Tools/scripts directory of your Python distribution. IDLE requires Tkinter. 


In addition, there are tools that will give you a more powerful and fancy Python 
interactive interpreter. One example is [Python, which is available at 
http://ipython.scipy.org/. 


1.2 Lexical matters 


1.2.1 Lines 


e Python does what you want it to do most of the time so that you only have to add 
extra characters some of the time. 

e Statement separator is a semi-colon, but is only needed when there is more than 
one statement on a line. And, writing more than one statement on the same line is 
considered bad form. 

e Continuation lines -- A back-slash as last character of the line makes the 


Page 15 


A Python Book 


following line a continuation of the current line. But, note that an opening 
"context" (parenthesis, square bracket, or curly bracket) makes the back-slash 
unnecessary. 


1.2.2 Comments 


Everything after "#" on a line is ignored. No block comments, but doc strings are a 
comment in quotes at the beginning of a module, class, method or function. Also, editors 
with support for Python often provide the ability to comment out selected blocks of code, 
usually with "##". 


1.2.3 Names and tokens 


e Allowed characters: a-z A-Z 0-9 underscore, and must begin with a letter or 
underscore. 
Names and identifiers are case sensitive. 
Identifiers can be of unlimited length. 
Special names, customizing, etc. -- Usually begin and end in double underscores. 
Special name classes -- Single and double underscores. 
o Single leading single underscore -- Suggests a "private" method or variable 
name. Not imported by "from module import *". 
o Single trailing underscore -- Use it to avoid conflicts with Python keywords. 
o Double leading underscores -- Used in a class definition to cause name 
mangling (weak hiding). But, not often used. 
e Naming conventions -- Not rigid, but: 
o Modules and packages -- all lower case. 
Globals and constants -- Upper case. 
Classes -- Bumpy caps with initial upper. 
Methods and functions -- All lower case with words separated by underscores. 
Local variables -- Lower case (with underscore between words) or bumpy 
caps with initial lower or your choice. 
o Good advice -- Follow the conventions used in the code on which you are 
working. 
e Names/variables in Python do not have a type. Values have types. 


QO: «OO 


1.2.4 Blocks and indentation 


Python represents block structure and nested block structure with indentation, not with 
begin and end brackets. 


The empty block -- Use the pass no-op statement. 


Benefits of the use of indentation to indicate structure: 
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e Reduces the need for a coding standard. Only need to specify that indentation is 4 
spaces and no hard tabs. 
e Reduces inconsistency. Code from different sources follow the same indentation 
style. It has to. 
e Reduces work. Only need to get the indentation correct, not both indentation and 
brackets. 
e Reduces clutter. Eliminates all the curly brackets. 
e If it looks correct, it is correct. Indentation cannot fool the reader. 
Editor considerations -- The standard is 4 spaces (no hard tabs) for each indentation level. 
You will need a text editor that helps you respect that. 


1.2.5 Doc strings 


Doc strings are like comments, but they are carried with executing code. Doc strings can 
be viewed with several tools, e.g. help (), obj.__doc__, and, in IPython, a question 
mark (?) after a name will produce help. 


A doc string is written as a quoted string that is at the top of a module or the first lines 
after the header line of a function or class. 


We can use triple-quoting to create doc strings that span multiple lines. 
There are also tools that extract and format doc strings, for example: 


e pydoc -- Documentation generator and online help system -- 
http://docs.python.org/lib/module-pydoc.html. 
e epydoc -- Epydoc: Automatic API Documentation Generation for Python -- 
http://epydoc.sourceforge.net/index.html 
e Sphinx -- Can also extract documentation from Python doc strings. See 
http://sphinx-doc.org/index.html. 
See the following for suggestions and more information on doc strings: Docstring 
conventions -- http://www.python.org/dev/peps/pep-0257/. 


1.2.6 Program structure 


e Execution -- def, class, etc are executable statements that add something to the 
current name-space. Modules can be both executable and import-able. 
Statements, data structures, functions, classes, modules, packages. 

Functions 

Classes 

Modules correspond to files with a "*.py" extension. Packages correspond to a 
directory (or folder) in the file system; a package contains a file named 
"init__.py". Both modules and packages can be imported (see section import 
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statement). 
e Packages -- A directory containing a file named "__init__.py". Can provide 
additional initialization when the package or a module in it is loaded (imported). 


1.2.7 Operators 


e See: http://docs.python.org/ref/operators.html. Python defines the following 
operators: 


Bs bess / // % 
<< 2S & | a a 
<— >= —— i <> 


The comparison operators <> and != are alternate spellings of the same operator. 
!= 1s the preferred spelling; <> is obsolescent. 
e Logical operators: 


and Oe is not lial 


e There are also (1) the dot operator, (2) the subscript operator [ ], and the 
function/method call operator (). 

e For information on the precedences of operators, see the table at 
http://docs.python.org/2/reference/expressions.html#operator-precedence, which 
is reproduced below. 

e For information on what the different operators do, the section in the "Python 
Language Reference" titled "Special method names" may be of help: 
http://docs.python.org/2/reference/datamodel.html#special-method-names 
The following table summarizes the operator precedences in Python, from lowest 
precedence (least binding) to highest precedence (most binding). Operators on the 
same line have the same precedence. Unless the syntax is explicitly given, 
operators are binary. Operators on the same line group left to right (except for 
comparisons, including tests, which all have the same precedence and chain from 
left to right -- see section 5.9 -- and exponentiation, which groups from right to 


left): 
Opegaleos Description 
lambda Lambda expression 
(ong Boolean OR 
and Boolean AND 
MOG 2 Boolean NOT 
alo; igf@ue aLial Membership tests 
ale, als} saKene Identity tests 
<, <=, >, >=, <>, !=, == Comparisons 
| Bitwise OR 
a Bitwise XOR 
& Bitwise AND 
eS Siabwies 
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at VG % 
remainder 
+X, —X 

~X 

K* 


x.attribute 

x [index] 

52 aonedlescsaimidesa] 

16 (Glisten blaleiqnetsys 5 a) 
(expressions...) 
[expressions...] 
{key:datum...} 
“expressions...- 


Addition and subtraction 
Multiplication, division, 


Positive, negative 
Bitwise not 
Exponentiation 
Attribute reference 
SUS CEA pr ton 

Syl akeraliayg} 

HunGcitaon  eallil 
Binding or tuple display 
List display 
Dictionary display 
String conversion 


e Note that most operators result in calls to methods with special names, for 
example __ add__,__sub__,__ mul __,, etc. See Special method names 
http://docs.python.org/2/reference/datamodel.html#special-method-names 
Later, we will see how these operators can be emulated in classes that you define 
yourself, through the use of these special names. 


1.2.8 Also see 


For more on lexical matters and Python styles, see: 


e Code Like a Pythonista: Idiomatic Python -- 
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html. 

e Style Guide for Python Code -- http://www.python.org/dev/peps/pep-0008/ 

e The flake8 style checking program. See https://pypi.python.org/pypi/flake8. Also 
see the pylint code checker: https://pypi.python.org/pypi/pylint. 


1.2.9 Code evaluation 


Understanding the Python execution model -- How Python evaluates and executes your 
code. 


Evaluating expressions. 


Creating names/variables -- Binding -- The following all create names (variables) and 
bind values (objects) to them: (1) assignment, (2) function definition, (3) class definition, 
(4) function and method call, (5) importing a module, ... 


First class objects -- Almost all objects in Python are first class. Definition: An object is 
first class if: (1) we can put it in a structured object; (2) we can pass it to a function; (3) 
we can return it from a function. 


References -- Objects (or references to them) can be shared. What does this mean? 


e The object(s) satisfy the identity test operator is. 
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e The built-in function id() returns the same value. 

e The consequences for mutable objects are different from those for immutable 
objects. 

e Changing (updating) a mutable object referenced through one variable or 
container also changes that object referenced through other variables or 
containers, because it is the same object. 

e del() -- The built-in function del () removes a reference, not (necessarily) the 
object itself. 


1.3 Statements and inspection -- preliminaries 


print -- Example: 


print obj 
pram “one, “ewoll,, three! 


for: -- Example: 


Stun = Vaan “bb “cer | 
ioe sheen shiay Swi S 
print item 


Learn what the type of an object is -- Example: 


type (obj) 


Learn what attributes an object has and what it's capabilities are -- Example: 


elie (Koloya)) 
value = "a message" 
dir (value) 


Get help on a class or an object -- Example: 


help (str) 

help ( wie ) 

value = "abc" 
help (value) 

help (value.upper) 


In [Python (but not standard Python), you can also get help at the interactive prompt by 
typing "?" and "??" after an object. Example: 


In [48]: a= '! 

In [49]: a.upper? 

Type: wake nes ncr EOnmom ame Elec 

String Form:<built-in method upper of str object at 0x7/f1c426e0508> 
Docst ring: 

S.upper() -> string 
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Return a copy of the string S converted to uppercase. 


1.4 Built-in data-types 


For information on built-in data types, see section Built-in Types -- 
http://docs.python.org/lib/types.html in the Python standard documentation. 


1.4.1. Numeric types 
The numeric types are: 


e Plain integers -- Same precision as a C long, usually a 32-bit binary number. 
e Long integers -- Define with 100L. But, plain integers are automatically 
promoted when needed. 
e Floats -- Implemented as a C double. Precision depends on your machine. See 
sys.float_info. 
e Complex numbers -- Define with, for example, 35 or complex (3.0, 2.0). 
See 2.3.4 Numeric Types -- int, float, long, complex -- 
http://docs.python.org/lib/typesnumeric.html. 


Python does mixed arithmetic. 


Integer division truncates. This is changed in Python 3. Use float (n) to force coercion 
to a float. Example: 


ina ese ey eS 

Tn [9s tb ="5 

iia, (lO) e ey 4/19 

Orbe UCN] 10) # possibly wrong? 
iat (ili) Ss icles ten) 7 Js 

(Oyehe ALE Shes) 


Applying the function call operator (parentheses) to a type or class creates an instance of 
that type or class. 


Scientific and heavily numeric programming -- High level Python is not very efficient for 
numerical programming. But, there are libraries that help -- Numpy and SciPy -- See: 
SciPy: Scientific Tools for Python -- http://scipy.org/ 


1.4.2 Tuples and lists 
List -- A list is a dynamic array/sequence. It is ordered and indexable. A list is mutable. 
List constructors: [], List (). 


range () and xrange(): 
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e range (n) creates a list of n integers. Optional arguments are the starting integer 
and a stride. 
e xrange is like range, except that it creates an iterator that produces the items 
in the list of integers instead of the list itself. 
Tuples -- A tuple is a sequence. A tuple is immutable. 


Tuple constructors: () , but really a comma; also tuple (). 
Tuples are like lists, but are not mutable. 


Python lists are (1) heterogeneous (2) indexable, and (3) dynamic. For example, we can 
add to a list and make it longer. 


Notes on sequence constructors: 


e Toconstruct a tuple with a single element, use (x, ) ; a tuple with a single 
element requires a comma. 
e Youcan spread elements across multiple lines (and no need for backslash 
continuation character "\"). 
e Acomma can follow the last element. 
The length of a tuple or list (or other container): len (mylist). 


Operators for lists: 


e Try: ie eee, ee ea, (ce ee eee, etc. 
e Comparison operators: <, ==, >=, etc. 
e Test for membership with the in operator. Example: 


Dm eles ae ly 2 = Ssh 
aera cone syerel 
Owe ITs ill, 22. S34 


ia [Pls 22 ais 2. 
Our Gils Teue 

ia [eos 44 an 2 
Out [80]: False 


Subscription: 


e Indexing into a sequence 

e Negative indexes -- Effectively, length of sequence plus (minus) index. 

e Slicing -- Example: data [2:5]. Default values: beginning and end of list. 

e Slicing with strides -- Example: data[::2]. 
Operations on tuples -- No operations that change the tuple, since tuples are immutable. 
We can do iteration and subscription. We can do "contains" (the in operator) and get the 
length (the Len () operator). We can use certain boolean operators. 


Operations on lists -- Operations similar to tuples plus: 


e Append -- mylist.append(newitem). 
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e Insert --mylist.insert (index, newitem). Note on efficiency: The 
insert method is not as fast as the append method. If you find that you need 
to do a large number of mylist.insert (0, obj) (that is, inserting at the 
beginning of the list) consider using a deque instead. See: 
http://docs.python.org/2/library/collections.html#collections.deque. Or, use 
append and reverse. 

e Extend --mylist.extend(anotherlist). Also can use + and +=. 

e Remove --mylist.remove (item) andmylist.pop(). Note that 
append () together with pop () implements a stack. 

e Delete -- del mylist [index]. 

e Pop -- Get last (right-most) item and remove from list -- mylist.pop(). 

List operators -- +, *, etc. 


For more operations and operators on sequences, see: 
http://docs.python.org/2/library/stdtypes.html#sequence-types-str-unicode-list-tuple-byte 
array-buffer-xrange. 


Exercises: 


e Create an empty list. Append 4 strings to the list. Then pop one item off the end 
of the list. Solution: 


ayy k25 a = ||| 

Ina [26 a.append('aaa') 

In 27]: a.append(‘bbb") 

In [28]: a.append('ccc') 

In [29]: a.append('ddd') 

Iba || SOS jovestioe 

[versie 7 Velole 7, Viele! Uiclolely || 
iar | Syab a.pop () 

Owe [sal Ve iloly 


e Use the for statement to print the items in the list. Solution: 


Iba SA |e stove ahieeiit aliat eye 
seen print item 


ccc 


e Use the string join operation to concatenate the items in the list. Solution: 


Him, ISS MN" = sjosne(a) 
Cie [Ssh s > Yevetey || \isiolai|| | tekere! 


e Use lists containing three (3) elements to create and show a tree: 


ina Swi "b= tbh Nome, None] 
ine iesly tes — sec None, Nome 
im [sols soot = Vaal; bp tei 
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iia P20) 

ia [401] 

In [40]: def show_tree(t): 
aeecenene Ihe Save. ee 
Seat return 
aegis jeakaje ie C)] 
jah show_tree(t[1]) 
foaiie ae show_tree(t[2]) 

In [41]: show_tree (root) 

aa 

bb 

ee 


Note that we will learn a better way to represent tree structures when we cover 
implementing classes in Python. 


1.4.3 Strings 


Strings are sequences. They are immutable. They are indexable. They are iterable. 


For operations on strings, see http://docs.python.org/lib/string-methods.html or use: 


Se Ine lljsy (ic 1) 


Or: 


Se Glatig (ellos ) 


String operations (methods). 
String operators, e.g. +, <, <=, ==, etc.. 
Constructors/literals: 


e Quotes: single and double. Escaping quotes and other special characters with a 
back-slash. 

e Triple quoting -- Use triple single quotes or double quotes to define multi-line 
strings. 

e str() -- The constructor and the name of the type/class. 

e ‘'aSeparator'.join(aList) 

e Many more. 

Escape characters in strings -- \t, \n, \\, etc. 


String formatting -- See: 
http://docs.python.org/2/library/stdtypes.html#string-formatting-operations 


Examples: 


In [18]: name = 'dave' 
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owe EN esac = 215 

Toe (20: wacton = 3.45 

im (20): prince "Names ss (sizer cdl actor so24h" 4 (mame, (sizer, 
EACEOr, ) 

Name: dav Suzece Zo backor: 3.4500 

In [25]: print 'Name: %s Size: %d Factor: %08.4f' % (name, size, 
IeVCNE@ue 

Name: dave Size: 25 Factor: 003.4500 


If the right-hand argument to the formatting operator is a dictionary, then you can 
(actually, must) use the names of keys in the dictionary in your format strings. Examples: 


In [115]: values = {'vegetable': 'chard', 'fruit': 'nectarine'} 
In [116]: 'I love S(vegetable)s and I love %(fruit)s.' % values 
Out ille| > “2 Hove ‘chard and 2 love nectarine.* 


Also consider using the right justify and left justify operations. Examples: 
MW SiC Ling. ejuUSsic (20) Bimysicicimeg.ljusie (20, "3k 


In Python 3, the str. format method is preferred to the string formatting operator. 
This method is also available in Python 2.7. It has benefits and advantages over the string 
formatting operator. You can start learning about it here: 
http://docs.python.org/2/library/stdtypes.html#string-methods 


Exercises: 


e Use a literal to create a string containing (1) a single quote, (2) a double quote, (3) 
both a single and double quote. Solutions: 


"Some 'quoted' text." 
'"Some "quoted" text.' 
"Some "quoted" \'extra\' text.' 


e Write a string literal that spans multiple lines. Solution: 


PME ols; (shesse al sqye; 
spans several lines 
because it is a little long. 


e Use the string join operation to create a string that contains a colon as a 
separator. Solution: 


[> Cemecine = || 


>>> content.append('finch') 
>>> content.append('sparrow') 
2] WOMeESine q elejosiarel (| Melee lise )) 
>>> content.append('jay') 


>>> contentstr = ':'.Jjoin(content) 
SS joiesine eomeSine sieie 
finch: sparrow:thrush: jay 


e Use string formatting to produce a string containing your last and first names, 


Page 25 


A Python Book 


separated by a comma. Solution: 


>>> first = 'Dave' 
>>> last = 'Kuhlman' 
See se SS esi eas ea ileisie acanesie ) 


ee jeugauane, sacl Il 
Kuhlman, Dave 


Incrementally building up large strings from lots of small strings -- the old way -- Since 
strings in Python are immutable, appending to a string requires a re-allocation. So, it is 
faster to append to a list, then use join. Example: 


leiay UPASh Ilo esievelkskGuce alli 

im [2o))s stelase append ( hane qi!) 
in (27> strict sappendh( lame 2”) 
Im [28 strilust append (Uitmer 4 3)) 
May (AC) | 2 Sicie = V iat! 2 ayepation (tical sete) 
ial || Si@) |S joweshote shai 

Line #1 

Line #2 

Line #3 


Incrementally building up large strings from lots of small strings -- the new way -- The 
+= operation on strings has been optimized. So, when you do this str1 += str2, 
even many times, it is efficient. 


The translate method enables us to map the characters in a string, replacing those in 
one table by those in another. And, the maket rans function in the st ring module, 
makes it easy to create the mapping table: 


SIMONE FSG TLIANS | 


def test(): 
a = 'axbycz' 
t = string.maketrans('abc', '123"') 
jOneaLinie, Gl 
prince a-translate(t) 


test () 


1.4.3.1 The new string.format method 


The new way to do string formatting (which is standard in Python 3 and perhaps 
preferred for new code in Python 2) is to use the string. format method. See here: 


e  http://docs.python.org/2/library/stdtypes.html#str.format 

e http://docs.python.org/2/library/string.html#format-string-syntax 

e http://docs.python.org/2/library/string.html#format-specification-mini-language 
Some examples: 
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Metal) LAL BV eleiei {il ielole; [0 ero” PIL) elekel Sarena (Vo VA) 
Oibhe [LIS Yelever Wy Jololey sec tere way elelcl\ 

im [Als “numbers -COeOody lok". terme (25)) 

Que Ze “mumber. 0002S ok” 

In [4]: ‘nl: {numl} n2: {num2}'.format (num2=25, numl=100) 
Que 4c ako nee Se 

ria (Si) 8 ia {numi > m22 ynum2; again: {numil}' format (mum2—2 5), 
num1=100) 

Owe Sills Maile IO) imzis Be; ereeistionS Aloo 

In [6 "number: TOG) Rok. Sif onsets (25) 

Otel Ss Vinbileyeias OWOZS ole’ 

In [7]: values = {'name': 'dave', '‘hobby': ‘birding'} 

In [8]: ‘user: {name} activity: {hobby}'.format (**values) 
CUES “Users dave Vactaivirye barccdimg. 


1.4.3.2 Unicode strings 


Representing unicode: 


in Pol: a = uUhalcd" 

iin (7) Ss et 

OuUEloyi usabcol 

In [98]: b = unicode ('efgh') 
iat |p SS) sie) 

ue oo ero hi 


Convert to unicode: a_string.decode (encoding) . Examples: 


mae lO Talocdt decode (Wer s) 

Owe POZA Ss qui! zvoyexe ¥ 

DEvay || L105) ] 

In [104] "abcd'.decode (sys.getdefaultencoding() ) 
Cue (l04) = uVabed" 


Convert out of unicode: a_unicode_string.encode (encoding) . Examples: 


Tiel PALO |S se) = Tol eiloyetelY 
in [08] = avencode (“uet—s") 


OME 0S]. saloed 
In [109]: a.encode(sys.getdefaultencoding() ) 
Ome 09] = abe! 


im [LLO)s bb =u Sel \xe7uk! 
in pene Wosencode (tiem) 
Selcuk 


Test for unicode type -- Example: 


In [122]: import types 

rn AS | a — aloe cd 

In [124]: type(a) is types.UnicodeType 
Ou (124) > true 

iia |) 25] 
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In [126]: type(a) is type(u'') 
Ou Ae). feus 


Or better: 


in (27 | asinistance (a, Umaeode)) 
Oue | e27 | tne 


An example with a character "c" with a hachek: 


tia [LoS 8 iene =} Viena waesical eed \o<e} 7! 
In [136]: name.decode('utf-8"') 

OMe USIo | Ss We Wyeia WWiesicd WOO WY 

stan Welest yes 

ia PLS} |) 8 Sia inven) 

Oued [bers | sib 

In [139]: len(name.decode ('utf-8"')) 
Ovune (ab sye/ 3° LAL 


You can also create a unicode character by using the unichr () built-in function: 


tia PAs ee Vever oe ulinisielloa (IL 7/0) ae Mole) 
ira Sea: 

Our IS: Ulaa\xaabl! 

im (ole b= anencode (“ubi=3) 

Tera ess Wo 

OwMEiV|e Yas vwe2 \sxzeilelo! 

IEG S|) 8 joueakiene, Io) 

aa*bb 


Guidance for use of encodings and unicode -- If you are working with a multibyte 
character set: 


1. Convert/decode from an external encoding to unicode early 
(my_string.decode (encoding) ). 
2. Do your work in unicode. 
3. Convert/encode to an external encoding late 
(my_string.encode (encoding) ). 
For more information, see: 


e Unicode In Python, Completely Demystified -- http://farmdev.com/talks/unicode/ 
e PEP 100: Python Unicode Integration -- 
http://www.python.org/dev/peps/pep-0100/ 
e Inthe Python standard library: 
o codecs -- Codec registry and base classes -- 
http://docs.python.org/2/library/codecs.html#module-codecs 
o Standard Encodings -- 
http://docs.python.org/2/library/codecs.html#standard-encodings 
If you are reading and writing multibyte character data from or to a file, then look at the 


Page 28 


A Python Book 


codecs.open() in the codecs module -- 
http://docs.python.org/2/library/codecs.html#codecs.open. 


Handling multi-byte character sets in Python 3 is easier, I think, but different. One hint is 
to use the encoding keyword parameter to the open built-in function. Here is an 
example: 


def test(): 
Mt We — open (alm tne sts. | Ws 
outfile = open('outfilel.txt', 
for line in infile: 
line = line.upper () 
outfile.write (line) 
dees CARS SIen()) 
outfile.close() 


7 encoding="uer—s™ ) 
w', encoding='utf-8') 


1 
J 


test () 


1.4.4 Dictionaries 


A dictionary is a collection, whose values are accessible by key. It is a collection of 
name-value pairs. 


The order of elements in a dictionary is undefined. But, we can iterate over (1) the keys, 
(2) the values, and (3) the items (key-value pairs) in a dictionary. We can set the value of 
a key and we can get the value associated with a key. 


Keys must be immutable objects: ints, strings, tuples, ... 


Literals for constructing dictionaries: 


ch = 
d2 = {keyl: valuel, key2: value2, } 


Constructor for dictionaries -- dict () can be used to create instances of the class dict. 
Some examples: 


olivene (4) Von) oe Vico a) Shi) 

Cue tone: 27) two 2) Se iatemisn())) 

auct ({ one's 2, “two's St eateritems ())) 

Cliche (aalor( (enor Ye Vicor) (25 Sh) ))) 

ance ()tewol 3] 7° one > 2) 

dict (one=2, two=3) 

der Citone!.. Stwor) e207 ae) sone 9 sine (27 Bl) 


For operations on dictionaries, see http://docs.python.org/lib/typesmapping.html or use: 


>>> help({}) 


Or: 
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Ss chief }) 


Indexing -- Access or add items to a dictionary with the indexing operator [ 


]. Example: 


itiay | ILiO7) -B yelatehe dl ft 

im [03] 7s caced | “mame |) — "dane 

ine (MOA diucel |(category || = 38 

litany (| ALOSy || selateieiL 

Our [05] {'category': 38, 'name': 'dave'} 


Some of the operations produce the keys, the values, and the items (pairs) in a dictionary. 


Examples: 
Ieiak [ssi B cla sieve, ae able oles So ee 2s) 
In [44]: d.keys() 
Out [44]: ['aa' Molen || 
In [45]: d.values () 
Omics (AS) 8 |plibil., 2273) 
In [46]: d.items() 
OueTAols siiCaal il) (bbw 222), 


When iterating over large dictionaries, use methods iterkeys (), itervalues(), 


and iteritems (). Example: 


dat (47 e 

Tee RAO aaa aS ison 2 

In [48]: for key in d.iterkeys(): 
Atoro oe print key 


aa 
bb 


To test for the existence of a key in a dictionary, use the in operator or the 
mydict.has_key (k) method. The in operator is preferred. Example: 


>> Ge | eomakvo! = iil Veucumoen ts soar 
>>> k = 'tomato' 

SS fe akial Cl 

True 

>>> d.has_key (k) 

True 


You can often avoid the need for a test by using method get. Example: 


SSS a tomato! Oil Veuieumbem "sso 2) 
[=> dedern (tomato, —)) 


> degeu (Vehbanchiy 1) 


>>> if d.get('eggplant') is None: 
foneabiic  Viqal Sisson] Y 
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missing 


Dictionary "view" objects provide dynamic (automatically updated) views of the keys or 
the values or the items in a dictionary. View objects also support set operations. Create 
views with mydict.viewkeys (),mydict.viewvalues (), and 
mydict.viewitems (). See: 
http://docs.python.org/2/library/stdtypes.html#dictionary-view-objects. 


The dictionary set default method provides a way to get the value associated with a 
key from a dictionary and to set that value if the key is missing. Example: 


ra OG | ara 

Ome woe) s 1 

ines sae seuderowitt (Vcc! Ss) 
Otte [les § SS} 

Deva {LALO @)]) 3) at 

Oneie [ANOS Seen! Ss 3y8) j 

Disa [IONS sks Sreimolenesnlle (ice 4) 
Oure ELLOS 3s 

erage oleae || eat 

@Uie [abil | Seal S38) 8) 


Exercises: 


e Write a literal that defines a dictionary using both string literals and variables 
containing strings. Solution: 


>>> first = 'Dave' 

>>> last = 'Kuhlman' 

>> iets Chics = (seaiesiee ies, Wnilwale ye Voices ike | 
22> (icine inves _CuLGie 

{*Dave’: “Kuhlman”, "“Elvis': “Presiley”™ | 


e Write statements that iterate over (1) the keys, (2) the values, and (3) the items in 
a dictionary. (Note: Requires introduction of the for statement.) Solutions: 


oe Gh aa A otis 22276 VEeN ee 839) 

>>> for key in d.keys(): 
print key 

aa 

ce 

bb 

>>> for value in d.values(): 
print value 

ae 

338 

Ze 


> hoi A eem aba (de ttemse( ie 
print item 
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(Saat, ial) 

(ee Sis) 

(Hb 222) 

>>> for key, value in d.items(): 
‘ print key, '::', value 

aa allele 

ce 333 

bb 222 


Additional notes on dictionaries: 


You can use iterkeys (), itervalues(),iteritems () to obtain 


iterators over keys, values, and items. 


A dictionary itself is iterable: it iterates over its keys. So, the following two lines 


are equivalent: 


One be alia) WgyAD aie | jouealinne 
for k im myDict- 1bterkeys(() 


jouesove, IS 


The in operator tests for a key in a dictionary. Example: 


In [52]: mydict = {'peach': 'sweet', 
In [53]: key = 'peach' 
In [o4]/2 if key in mydict: 
tue tonstat print mydict [key] 
sweet 


‘lemon': 'tangy'} 


1.4.5 Files 


Open a file with the open factory method. Example: 


im 28s ££ — open 'mylog., cst vw") 
In [29]: f.write('message #1\n') 
In [30]: f.write('message #2\n") 
In [31]: f.write('message #3\n') 
iar [Sas te e CuLOSS tO) 
Tm [SSie fo— fale Umnydog.txt* , is") 
im (34) tor lame an i: 
seers print line, 
message #1 
message #2 
message #3 
im [Sel taelkoser() 
Notes: 


e Use the (built-in) open (path, mode) function to open a file and create a file 
object. You could also use file (), but open () is recommended. 
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e A file object that is open for reading a text file supports the iterator protocol and, 
therefore, can be used in a for statement. It iterates over the lines in the file. This 
is most likely only useful for text files. 

e open isa factory method that creates file objects. Use it to open files for reading, 
writing, and appending. Examples: 


infile = open('myfile.txt', ‘'r') # open for reading 
outfile = open('myfile.txt', 'w') # open for (over-) 
writing 

log = open('myfile.txt', 'a') open for 
appending to existing content 


e When you have finished with a file, close it. Examples: 


abiane ILS elo (0) 
Out ileTelose ()) 


e Youcan also use the with: statement to automatically close the file. Example: 


with open('tmpOl.txt', 'r') as infile: 
1e@pe 2 alia) aligeaL ilies 
(NCH ep 


The above works because a file is a context manager: it obeys the context 
manager protocol. A file has methods ___enter___ and ___exit__, and the 
___exit__ method automatically closes the file for us. See the section on the 
with: statement. 

e Inorder to open multiple files, you can nest with: statements, or use a single 
with: statement with multiple "expression as target" clauses. Example: 


def test (): 
# 
# use multiple nested with: statements. 
Watt heoOpeme( {smell Mrs le te sate) te!) Wars) salt ike 
Wien open ( empmoukiile sexe, ‘Sw! ) as ouciale: 
jzowe  Ibsiete sisal alietse al lleye 
outfile.write('line: %Ss' % 


line.upper () ) 
joneabiahes alana al dlrs 
jouealione, ibhesaat ike 


# use a single with: statement. 
Welly Cyersio( Vasqvaiil emi ere Vp ie) eis) alionesi ley \ 
open (“tmplouttile sexe’, 'w') as outtiles 
for dane ain mealies 
outfile.write('line: %s' % line.upper()) 
print infile 
print outfile 


test () 


e file is the file type and can be used as a constructor to create file objects. But, 
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open is preferred. 
Lines read from a text file have a newline. Strip it off with something like: 
line. esicieijys (" \ia" )F 
For binary files you should add the binary mode, for example: rb, wb. For more 
about modes, see the description of the open () function at Built-in Functions -- 
http://docs.python.org/lib/built-in-funcs.html. 
Learn more about file objects and the methods they provide at: 2.3.9 File Objects 
-- http://docs.python.org/2/library/stdtypes.html#file-objects. 
You can also append to an existing file. Note the "a" mode in the following example: 


A Python Book 


iat |S) e 
lin [240)]| 2 
nEiay, (Sab es 
iia [482] 2 
iia 4s})) 
message 
message 
message 
message 
alistaln ele 


1g = oyersia {( imho - 
f.write('message #4\n"') 


f.close() 


£ = file('mylog. 
for dime in) if: 
peine dine, 


f.close() 


tesco yal) 


IESE ae vag 


For binary files, add "b" to the mode. Not strictly necessary on UNIX, but needed on MS 
Windows. And, you will want to make your code portable across platforms. Example: 


iia |) 2 import zipfile 

Jia {L683} outfile = open('tmpl.zip', '‘'whb') 

In [64 zfile = zipfile.ZipFile(outfile, 'w', zipfile.ZIP_DEFLATE 

In [65]: zfile.writestr('entryl', 'my heroes have always been 

cowboys') 

ita 6S zfile.writestr('entry2', ‘and they still are it seems') 

iim [67 zfile.writestr('entry3', 'sadly in search of and') 

Wi | es zfile.writestr('entry4', 'on step in back of") 

lial |X6)'$} 

ina | 7@ zfile.writestr('entry4', 'one step in back of') 

im [7 zfile.writestr('entry5', 'themselves and their slow moving 

ways') 

Absay || 22 zfile.close() 

ial {P73} outfile.close() 

eral MS 

$ 

S Unzip Sle temple zip 

Archive: tmpl.zip 

Length Method Size Ratio Date Time CRESS 2 Name 
34 Defl:N 36> +63 (Os—=29-03 17204" reb7do2i ecnienyal 
27 Defl:N 29° 72) is=29-038" 07  Mdagesa venery 2 
ZAP Wisse IL BIN ZA oc 0S = 2908 Wi O7 = siecle foal semiciey3 
18 Defl:N 20 =103 Oa=29-08 1720s “dbolsze6 sentry4 
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19 Defl1l:N Zo Osa 2o Oem OSs las aece wena 
37 Defl:N Se) So WS 2910s by SS) te ILS qOishe! euoneneyas) 
7) ToS) 85 6 files 


Exercises: 


e Read all of the lines of a file into a list. Print the 3rd and Sth lines in the file/list. 


Solution: 

in ioe i = open, (Vtmpil esc Pie) 
ia [oS lines = f.readlines() 
Iay |S) £.close() 
ia las lines 
Oue [58 Melee, Viesle oll, oreo ion! 6 “cher aw - 
Vaan “long \n 7.) hase an | 
Mal | SSS joueatione laos: || 2] 
brown 
In [61]: print lines[4] 
had 

More notes: 


e Strip newlines (and other whitespace) from a string with methods strip (), 
etme), and secu). 

e Get the current position within a file by using myfile.tell(). 

e Set the current position within a file by using myfile.seek (). It may be 
helpful to use os. SEEK_CUR and os. SEEK_END. For example: 
o f.seek(2, os.SEEK_CUR) advances the position by two 
o f£.seek(-—3, os.SEEK_END) sets the position to the third to last. 
o f£.seek (25) sets the position relative to the beginning of the file. 


1.4.6 Other built-in types 


Other built-in data types are described in section Built-in Types -- 
http://docs.python.org/lib/types.html in the Python standard documentation. 


1.4.6.1 The None value/type 


mom moo 


The unique value None is used to indicate "no value", "nothing", 
There is only one None value; in other words, it's a singleton. 


non-existence", etc. 


Use is to test for None. Example: 


Pe eka a) Nome 
SSS 
SS i ela use Nome: 


‘clear' 


jopenLione: 
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ClLeeuc 
>>> if flag is not None: 
pring. Vineiilo” 


Le 


1.4.6.2 Boolean values 
True and False are the boolean values. 


The following values also count as false, for example, in an if: statement: False, 
numeric zero, None, the empty string, an empty list, an empty dictionary, any empty 
container, etc. All other values, including True, act as true values. 


1.4.6.3 Sets and frozensets 
A set is an unordered collection of immutable objects. A set does not contain duplicates. 
Sets support several set operations, for example: union, intersection, difference, ... 


A frozenset is like a set, except that a frozenset is immutable. Therefore, a frozenset is 
hash-able and can be used as a key in a dictionary, and it can be added to a set. 


Create a set with the set constructor. Examples: 


>>> a = set () 

SS5 4 

sere (ip) 

>>> a.add('aa') 

SS als aiclel ( slo’ ) 

>>> a 

see (| vaat, “bbt]) 

SS joy = see ( (ily 22) 
> lb) 
sete Gly 22 |) 

SSS cS SSC AZ sisi) 
SS 105 bialaLoim (()) 
Seis alleys 2) |) 

SS be nieerseci om (ec) 
see G22) 


For more information on sets, see: Set Types -- set, frozenset -- 
http://docs.python.org/lib/types-set.html 


71.5 Functions and Classes -- A Preview 


Structured code -- Python programs are made up of expressions, statements, functions, 
classes, modules, and packages. 
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Python objects are first-class objects. 
Expressions are evaluated. 

Statements are executed. 

Functions (1) are objects and (2) are callable. 


Object-oriented programming in Python. Modeling "real world" objects. (1) 
Encapsulation; (2) data hiding; (3) inheritance. Polymorphism. 


Classes -- (1) encapsulation; (2) data hiding; (3) inheritance. 


An overview of the structure of a typical class: (1) methods; (2) the constructor; (3) class 
(static) variables; (4) super/subclasses. 


1.6 Statements 


1.6.1 Assignment statement 
Form -- target = expression. 
Possible targets: 


e Identifier 
e Tuple or list -- Can be nested. Left and right sides must have equivalent structure. 
Example: 


SSS x Vin Zo 
SSS (x, We @ A eee eS 
See ei, (lop )) De Cee SS) 
Soe ey, Bo Iie (2A 38) 


ly 22, Bs 


ieee 


This feature can be used to simulate an enum: 


Mia 22/8 yl 1K, MEDIUM, LARGE = range(l, 4) 
lerov fea Sh) S> Ine iB 

Ome Asie 

In [24]: MEDIUM 

One 2 


e Subscription of a sequence, dictionary, etc. Example: 


Ig) 
in 
isha Ab 
en 
in 
(Oibhes| fall 
Dry 
In 
isa) 


S 


: a = range(10) 


(Gay es Hes (Gey (Gs) |e) [= 


A Python Book 


Ove Molin (alate Me eet ly ii Oa! 

Iiey LISS loi Volos | = LION) 

ie igi Iolitee! |) = 2000 

Ia abst s is 

Out [18] {aa eS Viole OOO; ee 2 = 201010) 


e A slice of a sequence -- Note that the sequence must be mutable. Example: 


Hera els a = range (10) 

Asay | a 

Owe |Z Lio, ily Oy Sy 4, i 6, We 8, oF 

ne js aAca)| = ii 227 SiS A Shay, eile)| 

iin, |P4 a 

Out [4 Oe er al eee SiS Ae Oo (OOrun oy Oy dee donno) 
e Attribute reference -- Example: 

Pee lass MyClass: 


pass 


>>> anObj = MyClass () 


>>> anObj.desc = 'pretty' 
>>> prime ganOlla desc 
pretty 


There is also augmented assignment. Examples: 


>>> index = 0 
>>> index += 1 
>>> index += 5 
SSS alincles< qr Ge (5<)) 
1 
3} 


>>> index -= 
>>> index *= 


Things to note: 


e Assignment to a name creates a new variable (if it does not exist in the 
namespace) and a binding. Specifically, it binds a value to the new name. Calling 
a function also does this to the (formal) parameters within the local namespace. 

e In Python, a language with dynamic typing, the data type is associated with the 
value, not the variable, as is the case in statically typed languages. 

e Assignment can also cause sharing of an object. Example: 


Gag = AO 
obj2 = objl 


Check to determine that the same object is shared with id(obj) or the is 
operator. Example: 


In [23]: a = range(10) 

In [24] a 

Out [24] [Otpeatly eee Sirk yeeros mov ipa pamo | 
lay 925) || lon 

a PAG) S19) 
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OUR OG we 2 eS Ae Sane eerc, oil 
ere ee ee ONeill 2.8 Ss 

ia [233 Is 

Oui Silicy WO Wee S8ey, Oi Gre) Wieier Ol] 
ey P29 Io eh 

Oui T2SuS NO Ake 2 SSS ARTS, @O in er 2o)] 
ine WSO cas aks! a) 

Out [30]: True 

Alia IP SMEG Jonealions akeN(ei)) 7 alrol{(1o) 

SIMSTIZO SLOS7/S210 


e Youcan also do multiple assignment in a single statement. Example: 


im [SA “ae bb = 23 

lesay |S Sell bas el 

Owe Sse wes 

ia S43 Is 

One (SB ws 

derma [Soule 

aavedesroni|e 

im HSS) sar= bes ld 22 
ay |P Sie |S er aket Ie) 

Out [36]: True 


e You can interchange (swap) the value of two variables using assignment and 


packing/unpacking: 


>>> a = 111 

SS> b= 222 

>>> a, b=b, a 
SSS el 

2Za2, 

Sale) 

aa aL 


1.6.2 import statement 


Make module (or objects in the module) available. 


What import does: 


Evaluate the content of a module. 

Likely to create variables in the local (module) namespace. 

Evaluation of a specific module only happens once during a given run of the 
program. Therefore, a module is shared across an application. 

A module is evaluated from top to bottom. Later statements can replace values 
created earlier. This is true of functions and classes, as well as (other) variables. 
Which statements are evaluated? Assignment, class, def, ... 

Use the following idiom to make a module both run-able and import-able: 


aie name == ' main __': 
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# import pdb; pdb.set_trace() 
main () # or “test ()" 
defined in module 


or some other fumckr aon 


Notes: 


o The above condition will be true only when the module is run as a script and 


will not be true when the module is imported. 


o The line containing pdb can be copied any place in your program and 
un-commented, and then the program will drop into the Python debugger 


when that location is reached. 
Where import looks for modules: 


e sys.path shows where it looks. 
e There are some standard places. 


e Add additional directories by setting the environment variable PYTHONPATH. 


e Youcan also add paths by modifying sys.path, for example: 


import sys 
sys.path.insert (0, 


'/path/to/my/module') 


Packages need a file named ___init__.py. 


e Extensions -- To determine what extensions import looks for, do: 


>>> import imp 

>>> imp.get_suffixes () 

Poa syo Me Viale ensue, ue (Uinlevehbilie payroll 
I Wisiee ti sev p 4) 1 


Forms of the import statement: 


e import A-- Names in the local (module) namespace are accessible with the dot 


operator. 

e import A as B-- Import the module A, but bind the module object to the 
variable B. 

e import Al, A2-- Not recommended 

e from A import B 

e from A import Bl, B2 

e from A import B as C 

e from A import * -- Not recommended: clutters and mixes name-spaces. 

e from A.B import C-- (1) Possibly import object C from module B in 


package A or (2) possibly import module C from sub-package B in package A. 
e import A.B.C -- To reference attributes in C, must use fully-qualified name, 
for example use A.B.C.D to reference D inside of C. 
More notes on the import statement: 


e The import statement and packages -- A file named __init__.py is required in 
a package. This file is evaluated the first time either the package is imported or a 
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file in the package is imported. Question: What is made available when you do 
import aPackage? Answer: All variables (names) that are global inside the 
__ init__.py module in that package. But, see notes on the use of __all__ 
The import statement -- http://docs.python.org/ref/import.html 
e Theuseofif _ name__ == "__main__": -- Makes a module both 
import-able and executable. 
e Using dots in the import statement -- From the Python language reference manual: 
"Hierarchical module names:when the module names contains 
one or more dots, the module search path is carried out 
differently. The sequence of identifiers up to the last dot is used 
to find a package; the final identifier is then searched inside 
the package. A package is generally a subdirectory of a 
directory on sys.path that has a file __init__.py." 


See: The import statement -- http://docs.python.org/ref/import.html 
Exercises: 


e Import a module from the standard library, for example re. 

e Import an element from a module from the standard library, for example import 
compile from the re module. 

e Create a simple Python package with a single module in it. Solution: 

Create a directory named simplepackage in the current directory. 

Create an (empty) __ init__.py in the new directory. 

Create an simple. py in the new directory. 

Add a simple function name test 1 in simple.py. 

Import using any of the following: 


od ri 


>>> import simplepackage.simple 

>>> from simplepackage import simple 

>>> from simplepackage.simple import testl 

>>> from simplepackage.simple import testl as mytest 


1.6.3 print statement 


print sends output to sys. stdout. It adds a newline, unless an extra comma is 
added. 


Arguments to print: 


Multiple items -- Separated by commas. 

End with comma to suppress carriage return. 

Use string formatting for more control over output. 

Also see various "pretty-printing" functions and methods, in particular, pprint. 
See 3.27 pprint -- Data pretty printer -- 
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http://docs.python.org/lib/module-pprint.html. 
String formatting -- Arguments are a tuple. Reference: 2.3.6.2 String Formatting 
Operations -- http://docs.python.org/lib/typesseq-strings.html. 


Can also use sys. stdout. Note that a carriage return is not automatically added. 
Example: 


>>> import sys 
[> Sys Sseoourewrite (hello a) 


Controlling the destination and format of print -- Replace sys. stdout with an instance 
of any class that implements the method write taking one parameter. Example: 


IMIS Sys 


class Writer: 


aek ~ aimnatt  (Sselt, filename): 
self.out_file = file(file_name, ‘'a') 
def write(self, msg): 
self.out_file.write('[[%s]]' % msg) 
def close(self): 
self.out_file.close() 


Glee icecie ()) £ 
writer = Writer ('outputfile.txt') 
save_stdout = sys.stdout 


sys.stdout = writer 

peime helio? 

print 'goodbye' 

writer.close() 

# Show the output. 

tmp_file = file('outputfile.txt') 
sys.stdout = save_stdout 

content = tmp_file.read() 
tmp_file.close() 

print content 


test () 


There is an alternative form of the print statement that takes a file-like object, in 
particular an object that has a write method. For example: 


bia |feib outfile = open('tmp.log', 'w') 
ae (2 print >> outfile, 'Message #1' 
im [8 print >> outfile, 'Message #2' 
itiae (PZ print >> outfile, 'Message #3' 
iba ES) outfile.close() 
Ia (16 
Time [6 inte — open(*tmp.log*, ‘1 ) 
Aig’ [Ly PO dame wlliny aera et 
fexesbiotc, Vivebinie se Y,  ilavaer. esteieaujer(( / Vaal! )) 
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Line: Message #1 
Line: Message #2 
Line: Message #3 
In [8]: intile.close() 


Future deprecation warning -- There is no print statement in Python 3. There is a print 
built-in function. 


1.6.4 if: elif: else: statement 


A template for the if : statement: 


aie (Cionavelslicseril 

statements 
eller eonediteaonZ : 
statements 
elie eoncdaenoms': 
statements 
else: 

statements 


The elif and else clauses are optional. 


Conditions -- Expressions -- Anything that returns a value. Compare with eval () and 
exec. 


Truth values: 


e False -- False, None, numeric zero, the empty string, an empty collection (list 
or tuple or dictionary or ...). 
e True -- True and everything else. 


Operators: 
e andand or -- Note that both and and or do short circuit evaluation. 
e not 
e isandis not -- The identical object. Cf.a is bandid(a) == id(b). 


Useful to test for None, for example: 


ites es None: 


ik Se as mot, None: 


e inandnot in --Can be used to test for existence of a key in a dictionary or for 
the presence of a value in a collection. 
The in operator tests for equality, not identity. 
Example: 


Gia clays als S ipiom ss Ace 
Ses Veils! alia, el 
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True 
Se ele neice. aliay zl 
False 
SS sox Ts alian yo! 
False 
e Comparison operators, for example ==, ! =, <, <=, ... 


There is an if expression. Example: 


SSS El = VGiel! 
>>> b = 'bb' 
SSS ‘yes’ if a == b else 'no' 
SS 
' no J 
Notes: 


e The elif: clauses and the else: clause are optional. 

e Theif:,elif:,and else: clauses are all header lines in the sense that they 
are each followed by an indented block of code and each of these header lines 
ends with a colon. (To put an empty block after one of these, or any other, 
statement header line, use the pass statement. It's effectively a no-op.) 

e Parentheses around the condition in an if: or elif: are not required and are 
considered bad form, unless the condition extends over multiple lines, in which 
case parentheses are preferred over use of a line continuation character (backslash 
at the end of the line). 

Exercises: 


e Write an if statement with an and operator. 

e Write an if statement with an or operator. 

e Write an if statement containing both and and or operators. 
1.6.5 for: statement 
Iterate over a sequence or an "iterable" object. 


Form: 


TOI) ays 
iploek 


Iterator -- Some notes on what it means to be iterable: 


e An iterable is something that can be used in an iterator context, for example, in a 
for: statement, in a list comprehension, and in a generator expression. 

e Sequences and containers are iterable. Examples: tuples, lists, strings, 
dictionaries. 

e Instances of classes that obey the iterator protocol are iterable. See 
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http://docs.python.org/lib/typeiter.html. 

We can create an iterator object with built-in functions such as iter () and 
enumerate (). See Built-in Functions -- 
http://docs.python.org/lib/built-in-funcs.html in the Python standard library 
reference. 

Functions that use the yield statement, produce an iterator, although it's actually 
called a generator. 

An iterable implements the iterator interface and satisfies the iterator protocol. 
The iterator protocol: ___iter___() and next () methods. See 2.3.5 Iterator 
Types -- (http://docs.python.org/lib/typeiter.html). 


Testing for "iterability": 


If you can use an object in a for: statement, it's iterable. 
If the expresion iter (obj) does not produce a TypeError exception, it's 
iterable. 


Some ways to produce iterators: 


iter() andenumerate() -- See: 
http://docs.python.org/lib/built-in-funcs.html. 

some_dict.iterkeys(), some_dict.itervalues(), 

SCM Chie . 1icerLicSims ())k 

Use a sequence in an iterator context, for example in a for statement. Lists, 
tuples, dictionaries, and strings can be used in an iterator context to produce an 
iterator. 

Generator expressions -- Latest Python only. Syntactically like list 
comprehensions, but (1) surrounded by parentheses instead of square brackets and 
(2) use lazy evaluation. 

A class that implements the iterator protocol -- Example: 


class A(object): 

Get Sahimatcaaa(seillr) es 
selft.data = [11,22,33] 
self.idx = 0 

oleae Lo ahigeie  (UsSillae)) 2 
return self 

def next (self): 
if self.idx < len(self.data): 

x = self.data[self.idx] 
self.idx +=1 
return xX 
else: 
raise StopIteration 


def test(): 
al = IN) 
aejie oe alae ele 
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jouealione, > 


test () 


Note that the iterator protocol changes in Python 3. 

A function containing a yield statement. See: 

o Yield expressions -- 
http://docs.python.org/2/reference/expressions.html#yield-expressions 

o The yield statement -- 
http://docs.python.org/2/reference/simple_stmts.html#the-yield-statement 

Also see itertools module in the Python standard library for much more help 

with iterators: itertools — Functions creating iterators for efficient looping -- 

http://docs.python.org/2/library/itertools.html#module-itertools 


The for: statement can also do unpacking. Example: 


COONS) 


[25]: items = ['apple', 'banana', 'cherry', 'date'] 
[26]: for idx, item in enumerate (items): 

ev ekayice’s joueakiae broly zatset | rs (lake bse) Paley) 

appl 

banana 

eincraiay 

date 


The for statement can also have an optional else: clause. The else: clause is 
executed if the for statement completes normally, that is if a break statement is not 
executed. 


Helpful functions with for: 


e enumerate (iterable) -- Returns an iterable that produces pairs (tuples) 


containing count (index) and value. Example: 


SoS hom slide, Wakuie aim emumeiqake (als 227 Sah] \e: 
print idx, value 


@ wil 
22 
2°33 


e range([start,] stop[, step]) andxrange([start,] stopl[, 


step]). 


List comprehensions -- Since list comprehensions create lists, they are useful in for 
statements, although, when the number of elements is large, you should consider using a 
generator expression instead. A list comprehension looks a bit like a for: statement, but 
is inside square brackets, and it is an expression, not a statement. Two forms (among 


others): 


[i Ge) 80% & am riteralsle] 
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[i (x) tor x am witeralosle wit & (x) | 


Generator expressions -- A generator expression looks similar to a list comprehension, 
except that it is surrounded by parentheses rather than square brackets. Example: 


In [28 teens — | ‘appille’) “banana, ) “cherry “date? | 
In [29]: genl = (item.upper() for item in items) 
ne SO ey ore So ein gem: 
oy aera: joncakone US. Ve oe 
x: APPLE 
x: BANANA 
x: CHERRY 
x? DATE 
Exercises: 
e Write a list comprehension that returns all the keys in a dictionary whose 
associated values are greater than zero. 
o The dictionary: (RBI imeem pune te eet le ote 
o Solution: [x[0] for x in my_dict.iteritems() if x[1] > 
0] 
e Write a list comprehension that produces even integers from 0 to 10. Use a for 
statement to iterate over those values. Solution: 
for x an [ty for y an range (0) if y <2 —— Ol: 
jonenione, Voee sail ae Se 
e Write a list comprehension that iterates over two lists and produces all the 


combinations of items from the lists. Solution: 


ia | 9) a = range (4) 

ina [| 20) lol ear || Ihe 272 83) 

Ie |i 2ub sel 

Ome (Aas Oy sls 2 Sh 

Ia 22 b 

Owe 22 a 222 SS) 

ia [23 co] Ga y)) tome anal kom sy abn py 

Ten 2 lice ore te 

[(O, It), (0, 22), (0, 33), (i, 1), Ul, 22), Ul, 335 
(2, LI); Wp 22) 7 (2, SS)p (Sp 1), Wee 22a), (Sp S35] 


But, note that in the previous exercise, a generator expression would often be better. A 
generator expression is like a list comprehension, except that, instead of creating the 
entire list, it produces a generator that can be used to produce each of the elements. 


The break and continue statements are often useful in a for statement. See continue 
and break statements 


The for statement can also have an optional else: clause. The else: clause is 
executed if the for statement completes normally, that is if a break statement is not 
executed. Example: 
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ieee alive) uliay telaidelil 
ae shies = IL0N0) ¢ 


valuel = item 
break 
else: 
valuel = 'not found' 


print 'valuel:', valuel 


When run, it prints: 


Velluleits Inet  koumel 


1.6.6 while: statement 


Form: 


Whaghke ie omnes som: 
block 


The while: statement is not often used in Python because the for: statement is 
usually more convenient, more idiomatic, and more Pythonic. 


Exercises: 


e Write a while statement that prints integers from zero to 5. Solution: 


count = 0 

While count. < 5): 
count += 1 
Pring Count 


The break and continue statements are often useful in a while statement. See 
continue and break statements 


The while statement can also have an optional else: clause. The else: clause is 
executed if the while statement completes normally, that is if al break statement is not 
executed. 


1.6.7 continue and break statements 
The break statement exits from a loop. 


The continue statement causes execution to immediately continue at the start of the 
loop. 


Can be used in for: and while:. 


When the for: statement or the while: statement has an else: clause, the block in 
the else: clause is executed only if a break statement is not executed. 


Exercises: 
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e Using break, write a while statement that prints integers from zero to 5. 
Solution: 


count = 0 
while True: 
count += 1 
ile Keyonblane, ye 
break 
Prine count 


Notes: 
o A for statement that uses range () or xrange() would be better than a 
while statement for this use. 
e Using continue, write a while statement that processes only even integers 
from 0 to 10. Note: % is the modulo operator. Solution: 


count = 0 
while count << 0: 
count += i 
if count % 2 == 0: 
continue 
PLIne Coun 


1.6.8 try: except: statement 


Exceptions are a systematic and consistent way of processing errors and "unusual" events 
in Python. 


Caught and un-caught exceptions -- Uncaught exceptions terminate a program. 
The try: statement catches an exception. 
Almost all errors in Python are exceptions. 


Evaluation (execution model) of the t ry statement -- When an exception occurs in the 
try block, even if inside a nested function call, execution of the t ry block ends and the 
except clauses are searched for a matching exception. 


Tracebacks -- Also see the t raceback module: 
http://docs.python.org/lib/module-traceback.html 


Exceptions are classes. 
Exception classes -- subclassing, args. 


An exception class in an except: clause catches instances of that exception class and 
all subclasses, but not superclasses. 


Built-in exception classes -- See: 


e Module exceptions. 
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e Built-in exceptions -- http://docs.python.org/lib/module-exceptions.html. 
User defined exception classes -- subclasses of Exception. 


Example: 


ive: 
raise RuntimeError('this silly error') 
except RuntimeError, exp: 


° 


jorerkioje YI | [easi| i) Yes eee 


Reference: http://docs.python.org/lib/module-exceptions.html 


You can also get the arguments passed to the constructor of an exception object. In the 
above example, these would be: 


exp.args 


Why would you define your own exception class? One answer: You want a user of your 
code to catch your exception and no others. 


Catching an exception by exception class catches exceptions of that class and all its 
subclasses. So: 


except SomeExceptionClass, exp: 


matches and catches an exception if SomeExceptionClass is the exception class or a base 
class (superclass) of the exception class. The exception object (usually an instance of 
some exception class) is bound to exp. 


A more "modern" syntax is: 


except SomeExceptionClass as exp: 


So: 


class MyE(ValueError): 
pass 


wives 
raise MyE() 
except ValueError: 
print ‘caught exception’ 


will print "caught exception", because ValueError is a base class of MyE. 


Also see the entries for "EAFP" and "LBYL" in the Python glossary: 
http://docs.python.org/3/glossary.html. 


Exercises: 


e Write a very simple, empty exception subclass. Solution: 


class MyE (Exception): 
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pass 


e Write atry:except: statement that raises your exception and also catches it. 
Solution: 


leva 

raise MyE('hello there dave') 
Sexes Or Mbyh, ie 

pLink © 


1.6.9 raise statement 
Throw or raise an exception. 
Forms: 


e raise instance 
e raise MyExceptionClass (value) -- preferred. 
e raise MyExceptionClass, value 

The raise statement takes: 


An (instance of) a built-in exception class. 

An instance of class Exception or 

An instance of a built-in subclass of class Exception or 

An instance of a user-defined subclass of class Exception or 

One of the above classes and (optionally) a value (for example, a string or a 
tuple). 

See http://docs.python.org/ref/raise.html. 


For a list of built-in exceptions, see http://docs.python.org/lib/module-exceptions.html. 


The following example defines an exception subclass and throws an instance of that 
subclass. It also shows how to pass and catch multiple arguments to the exception: 


class NotsobadError (Exception) : 
pass 


ClGie iSite (P<) = 
Gry 
if x == 
raise NotsobadError('a moderately bad error', ‘not too 
bad') 
except NotsobadError, e: 
joueulians) Vidneiaove chaise ws! ss (excuccis,; )) 


test (0) 


Which prints out the following: 


Error args: ('a moderately bad error', 'not too bad') 
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Notes: 


e Inorder to pass in multiple arguments with the exception, we use a tuple, or we 
pass multiple arguments to the constructor. 
The following example does a small amount of processing of the arguments: 


class NotsobadError (Exception) : 
"""An exception class. 


ww 


def get_args(self): 
asi wiiein Ye hee VS airoskiow (isyeulis 4 clicrois),) 


def test (x): 
(eleva 
if x == 
raise NotsobadError('a moderately bad error', ‘not too 


bac") 
except NotsobadError, e: 
primME wNeror args: iijost)y =o (eageusargs 7 ) 


test (0) 


1.6.10 with: statement 


The with statement enables us to use a context manager (any object that satisfies the 
context manager protocol) to add code before (on entry to) and after (on exit from) a 
block of code. 


1.6.10.1 Writing a context manager 


A context manager is an instance of a class that satisfies this interface: 


class Context0l1 (object): 
def __enter___(self): 
pass 
def __exit__(self, exc_type, exc_value, traceback): 
pass 


Here is an example that uses the above context manager: 


class Context0l1 (object): 
def __enter_ (self): 
print 'in __enter__' 


return ‘some value or other' # usually we want to return 
self 
def __exit__(self, exc_type, exc_value, traceback): 
fone aban ae aLseWn ee Secchi ce Ny 
Notes: 
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e The__enter__ method is called before our block of code is entered. 
e Usually, but not always, we will want the __enter__ method to return self, 
that is, the instance of our context manager class. We do this so that we can write: 


with MyContextManager() as obj: 
pass 


and then use the instance (obj in this case) in the nested block. 

e The __exit__ method is called when our block of code is exited either 
normally or because of an exception. 

e If an exception is supplied, and the method wishes to suppress the exception (i.e., 
prevent it from being propagated), it should return a true value. Otherwise, the 
exception will be processed normally upon exit from this method. 

e Ifthe block exits normally, the value of exc_type, exc_value, and 
traceback will be None. 

For more information on the with: statement, see Context Manager Types -- 
http://docs.python.org/2/library/stdtypes.html#context-manager-types. 


See module context 1ib for strange ways of writing context managers: 
http://docs.python.org/2/library/contextlib.html#module-contextlib 


1.6.10.2 Using the with: statement 


Here are examples: 


# example 1 
with Context01(): 
joneakioye align’ loxeyely J 


# example 2 


with Context02() as a_value: 
joeakime, ation Joyexely! 
print 'a_value: "%s"" % (a_value, ) 


a_value.some_method_in_Context02 () 


# example 3 


with open(infilename, 'r') as infile, open(outfilename, 'w') as 
outfile: 
ioe sine) aliat aticnieslL bes 
line = line.rstrip() 


Q 


QUELT Ter write (ss \n! 2 lane supper() ) 


Notes: 


e Intheformwith ... as val, the value returned by the__enter__ 
method is assigned to the variable (val in this case). 

e Inorder to use more than one context manager, you can nest with: statements, 
or separate uses of of the context managers with commas, which is usually 
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preferred. See example 3 above. 


1.6.11 del 


The del statement can be used to: 


e Remove names from namespace. 

e Remove items from a collection. 
If name is listed in a global statement, then de1 removes name from the global 
namespace. 


Names can be a (nested) list. Examples: 


>>> dela 
So ceil vay) ie 


We can also delete items from a list or dictionary (and perhaps from other objects that we 
can subscript). Examples: 


Te ONG —aweteias-2 uleilh Winlae OO ee es SS 3y 
Itiat (ALO) Sjosestiave, cl 

(atc: SIME ices 88506 Ole 3). 22:2) 

iia |p Aba] Brelevik eli) oles! 

diay AL 2) )) Bieseakione xl 

(aikelet Geib iil hes Mielec se cysh eit 

ery peleSiile 

ine (iesisac= iii 2227, 383, 7444) 
im (14) spring a 

[i 2227 38375444) 

Tm [aksecledsscalhil 

ibiay | AlSy]) Sioiestiane “i 

[111, 333, 444] 


And, we can delete an attribute from an instance. Example: 


ia [yl eeilasise Av: 
pass 


ia (ie) ee = AG) 
ial (JES) Seyeoe = les 
Iiae  [-ZA10))) relishes (Ve) 


One lO adocw Smo duikew sah 
Aiay [|Z 1h | Bieweationcy era >< 
Ieee 


ine 22) clei ax 

iat |b Sill Siclsese (ey) 

Oe L238 Y= aelexe YS VL imerclistilie | 
Mio P22) Bioieationc, ele 


exceptions.AttributeError Traceback, (most necente Callie Nase) 
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/home/dkuhlman/al/Python/Test/<console> 


Aperiioubebrrorm: A ainstance has me) akerioute "sa! 


1.6.12 case statement 


There is no case statement in Python. Use the if: statement with a sequence of elif: 
clauses. Or, use a dictionary of functions. 


1.7 Functions, Modules, Packages, and Debugging 


1.7.1 Functions 


1.7.1.1 The def statement 
The def statement is used to define functions and methods. 


The def statement is evaluated. It produces a function/method (object) and binds it to a 
variable in the current name-space. 


Although the def statement is evaluated, the code in its nested block is not executed. 
Therefore, many errors may not be detected until each and every path through that code is 
tested. Recommendations: (1) Use a Python code checker, for example flake8 or 
pylint; (2) Do thorough testing and use the Python unittest framework. Pythonic 
wisdom: If it's not tested, it's broken. 


1.7.1.2 Returning values 
The return statement is used to return values from a function. 


The return statement takes zero or more values, separated by commas. Using commas 
actually returns a single tuple. 


The default value is None. 


To return multiple values, use a tuple or list. Don't forget that (assignment) unpacking 
can be used to capture multiple values. Returning multiple items separated by commas is 
equivalent to returning a tuple. Example: 


In (8]¢ def test (x, y): 
CUI Sees Sry, 7 a4 


iia (Shs ap iss = easie(S, 4) 
In (10]s print a 
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Ieiay ALdk |] 8 jeneakiowe Io 
16 


1.7.1.3 Parameters 


Default values -- Example: 


In fos]? det € (max—5) : 
i fektey ieee for val in range(max): 
seins joneshioke, yell 

Im [54] 2 t (3) 

0 

ik 

2) 

Time aoe eG) 

0 

i 

2 

3 

4 


Giving a parameter a default value makes that parameter optional. 


Note: If a function has a parameter with a default value, then all "normal" arguments 
must proceed the parameters with default values. More completely, parameters must be 
given from left to right in the following order: 


1. Normal arguments. 

2. Arguments with default values. 

3. Argument list (*args). 

4. Keyword arguments (**kwargs). 
List parameters -- *args. It's a tuple. 


Keyword parameters -- **kwargs. It's a dictionary. 


1.7.1.4. Arguments 


When calling a function, values may be passed to a function with positional arguments or 
keyword arguments. 


Positional arguments must placed before (to the left of) keyword arguments. 


Passing lists to a function as multiple arguments -- some_func (*aList) . Effectively, 
this syntax causes Python to unroll the arguments. Example: 


det anii(varcs,. sakwereesi) 
Ent angs,, “~kwargs)) 
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1.7.1.5 Local variables 


Creating local variables -- Any binding operation creates a local variable. Examples are 
(1) parameters of a function; (2) assignment to a variable in a function; (3) the import 
statement; (4) etc. Contrast with accessing a variable. 


Variable look-up -- The LGB/LEGB rule -- The local, enclosing, global, built-in scopes 
are searched in that order. See: http://www.python.org/dev/peps/pep-0227/ 


The global statement -- Inside a function, we must use global when we want to set 
the value of a global variable. Example: 


def fn(): 
global Some_global_variable, Another_global_variable 
Some_global_variable = 'hello' 


1.7.1.6 Other things to know about functions 


e Functions are first-class -- You can store them in a structure, pass them to a 
function, and return them from a function. 
e Function calls can take keyword arguments. Example: 


>>> test (size=25) 


e Formal parameters to a function can have default values. Example: 


>>> def test (size=0): 


Do not use mutable objects as default values. 
e Youcan "capture" remaining arguments with *args, and **kwargs. (Spelling 
is not significant.) Example: 


In [13]2 det test (size, *args, **kwargs) : 
Hittites print size 
esuscuas print args 

print kwargs 


in, |i tese(32a7 taa*;, Y‘bbo')) otherparam— "xyz." ) 


Ghareny Visto") 
{'otherparam': 'xyz'} 


e Normal arguments must come before default arguments which must come before 


keyword arguments. 
e A function that does not explicitly return a value, returns None. 
e Inorder to set the value of a global variable, declare the variable with global. 


Exercises: 
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e Write a function that takes a single argument, prints the value of the argument, 
and returns the argument as a string. Solution: 


-So der tice): 
jolienioie, U5e8 as 
return '[[%s 


e Write a function that takes a variable number of arguments and prints them all. 
Solution: 


Se OSI ie (eles) 5 
imGne elas) Alin’ euacyisys 
pain Vane 2s so varag, 


See se Vales “obi seicr) 


arg: aa 
glee) lelle) 
age ee 


e Write a function that prints the names and values of keyword arguments passed to 
it. Solution: 


>>> def t(**kwargs): 

for key in kwargs.keys(): 
seen print 'key: %s value: %s' % (key, 
kwargs[key], ) 


Se > ie (cue ill ice 2—= 22) 
key: argl value: 11 
key: arg2 value: 22 


1.7.1.7 Global variables and the global statement 
By default, assignment in a function or method creates local variables. 


Reference (not assignment) to a variable, accesses a local variable if it has already been 
created, else accesses a global variable. 


In order to assign a value to a global variable, declare the variable as global at the 
beginning of the function or method. 


If in a function or method, you both reference and assign to a variable, then you must 
either: 


1. Assign to the variable first, or 

2. Declare the variable as global. 
The global statement declares one or more variables, separated by commas, to be 
global. 
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Igiay {fil 
mana’ X = 3 
iia [2 def t(): 
joreakione, >.< 
im (3 
dears en) 
3 
In [4 def s(): 
xX =4 
iba 
tal piles 
Tia ES) Se) 
ia |) 6 ic) 
5) 
nay oles ee —) ob 
im eile der was 
eee global X 
Peas x= 5 
iar [Sy< 
ime LON EG) 
ial [LAOS tet) 
5) 
Abia [LAL def v(): 
Sean ree x = X 
Pane ree X = 6 
Cane return xX 
Iiak [10 7 || 
diag PAR TAB Aw (0) 
Traceback (most recent call last): 
File "<ipython console>", line 1, in <module> 
Pike Vw lpytehon Console>, | iine 27) ane 
UnboundLocalError: local variable 'X' referenced before assignment 
im (LSis Cer wi): 
eon oe global X 
sates x = X 
ects xX = 7 
me aoeoee BEC Ue x 
etary | 1.93] 
Ia |RSS tO) 
@ulie [ALS] 5 
iia || 210) xX 
the [20] qi 
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1.7.1.8 Doc strings for functions 


Add docstrings as a triple-quoted string beginning with the first line of a function or 
method. See epydoc for a suggested format. 


1.7.1.9 Decorators for functions 


A decorator performs a transformation on a function. Examples of decorators that are 
built-in functions are: @classmethod, @staticmethod, and @property. See: 
http://docs.python.org/2/library/functions.html#built-in-functions 


A decorator is applied using the "@" character on a line immediately preceeding the 
function definition header. Examples: 


class SomeClass (object): 


@classmethod 
Clene isle IibolCileisis (ells, eles) 
pass 
## HelloClass = classmethod(HelloClass) 


@staticmethod 
def HelloStatic(arg): 
pass 
## HelloStatic = staticmethod(HelloStatic) 


# 
# Define/implement a decorator. 
def wrapper (fn): 
def inner_fn(*args, **kwargs): 
foneabinic, eau 
mesulle = iil (sees, Ix ielecis)) 
joneakiones kee 
return result 
iste wlisia aliabaleue ictal 


# 
# Apply a decorator. 
@wrapper 
def fnl(msg) : 
pass 
## fnl = wrapper (fnl) 


Notes: 


e The decorator form (with the "@" character) is equivalent to the form 
(commented out) that calls the decorator function explicitly. 

e The use of classmethods and staticmethod will be explained later in the 
section on object-oriented programming. 

e A decorator is implemented as a function. Therefore, to learn about some specific 
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decorator, you should search for the documentation on or the implementation of 
that function. Remember that in order to use a function, it must be defined in the 
current module or imported by the current module or be a built-in. 

e The form that explicitly calls the decorator function (commented out in the 
example above) is equivalent to the form using the "@" character. 


1.7.2 lambda 
Use a lambda, as a convenience, when you need a function that both: 


e is anonymous (does not need a name) and 
e contains only an expression and no statements. 


Example: 
Aeiay {Lib inlay == hennilloyslen 2 ie oe (et ares ts ZI ares 
te 2 ‘em (4, S, 6) 
Owe |Z 32 
alot Salis 
lin (Sts sconaineic = ibenmlecla olog|, @eleecjeiey7s Vlloes Wesel aie ey Wee GY 5 (colo), 
category, ) 
tia [4 format ('pine tree', 'conifer') 
Out [4 "The "pine tree" is a "conifer".' 


A lambda can take multiple arguments and can return (like a function) multiple values. 
Example: 


lin [Vee a = ibeiilocle se, ws (Se yy = 4 (se, 9) 
IEiat (eH) I] 5 

iimae [PGA ani <4) 

Cure ells (Ip We, (Se 4) 


Suggestion: In some cases, a lambda may be useful as an event handler. 


Example: 


class Test: 
det init (Silk skies karst es 
Sells. manesie »—— ales 
self.last = last 
dek Pesu(seln, Formate): 


ww 


Test for lambdas. 
formatter is a function taking 2 arguments, first and last 


names. It should return the formatted name. 
wo 
msg = 'My name is %s' % (formatter(self.first, self.last),) 


prink msg 


def test(): 
t = Test('Dave', 'Kuhlman') 
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tl test Glamodal mirsty. iJraisitr 
eee (enmisiele, sesliesie, Ilene e 


ms) ws \igiiesic, ilesic,  )) )) 
eos o Gbastey, ate’stey: 5)) 


test () 


A lambda enables us to define "functions" where we do not need names for those 
functions. Example: 


arate [EAS sa all 
seen: Lambda a2 x7 
wee) bambb dar ex sss 41 2, 


In [46 

In [46]: a 

Ourl4o]= [<tunction “main ><lambda>>, <function ~ main .<lambda>>] 
irae | [PAs hiliese eer MOMS) 

Oui Ay ss 

Time PAS. alls) 

Outer 483) 2 36 


Reference: http://docs.python.org/2/reference/expressions.html#lambda 


1.7.3 Iterators and generators 
Concepts: 
iterator 


And iterator is something that satisfies the iterator protocol. Clue: If it's an iterator, 
you can use it ina for: statement. 


generator 


A generator is a class or function that implements an iterator, i.e. that implements the 
iterator protocol. 


the iterator protocol 
An object satisfies the iterator protocol if it does the following: 


o Itimplements a__iter__ method, which returns an iterator object. 

o Itimplements a next function, which returns the next item from the 
collection, sequence, stream, etc of items to be iterated over 

o Itraises the StopIteration exception when the items are exhausted and 
the next () method is called. 


yield 


The yield statement enables us to write functions that are generators. Such 
functions may be similar to coroutines, since they may "yield" multiple times and are 
resumed. 
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For more information on iterators, see the section on iterator types in the Python Library 
Reference -- http://docs.python.org/2/library/stdtypes.html#iterator-types. 


For more on the yield statement, see: 
http://docs.python.org/2/reference/simple_stmts.html#the-yield-statement 


Actually, yield is an expression. For more on yield expressions and on the next () 
and send() generator methods, as well as others, see: Yield expression -- 
http://docs.python.org/2/reference/expressions.html#yield-expressions in the Python 
language reference manual. 


A function or method containing a yield statement implements a generator. Adding the 
yield statement to a function or method turns that function or method into one which, 
when called, returns a generator, i.e. an object that implements the iterator protocol. 


A generator (a function containing yield) provides a convenient way to implement a 
filter. But, also consider: 


e The filter () built-in function 
e List comprehensions with an if clause 
Here are a few examples: 


def simplegenerator(): 


yield 'aaa' # Note 1 
yield 'bbb' 
yield ccc" 


def list_tripler(somelist): 
for item in somelist: 
item *= 3 
yield item 


Gef limit_iterator (somelist, max): 
hom Gscem ene Someissst < 
Sie Shine + ieis<e 
return # Note 2 
yield item 


Glene cissie Me 
forbes VaR ab a3 |@) 
it = simplegenerator () 


OMe sexsi Siar abet 
print item 


[OGRE YAN Cae Mae Ss au a eres SC) 
alist = range(5) 
ie = hate. eiesrjoulrsse (ei lLatshe))) 


neve skye Siok okie 
print item 

Oued: Si. Uy Pa be A 50 

alist = range (8) 
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ie == Ihab@alie siercwecicoue (ellivsic, 4) 
oie see Shiota 
print item 


prince: U4 val 9230 
it = simplegenerator () 
erm ye 


print it.next () # Note 3 
print it.next () 

print it.next () 

PINE Le next () 
except StopIteration, exp: # Note 4 


print 'reached end of sequence’ 


aie name == iiqeval iat S 

iESsie () 
Notes: 

1. The yield statement returns a value. When the next item is requested and the 
iterator is "resumed", execution continues immediately after the yield 
statement. 

2. We can terminate the sequence generated by an iterator by using a return 
statement with no value. 

3. To resume a generator, use the generator's next () or send() methods. 
send () is like next (), but provides a value to the yield expression. 

4. We-can alternatively obtain the items in a sequence by calling the iterator's 
next () method. Since an iterator is a first-class object, we can save it in a data 
structure and can pass it around for use at different locations and times in our 
program. 

1. When an iterator is exhausted or empty, it throws the StopIteration 


exception, which we can catch. 


And here is the output from running the above example: 


Ie 


i) 


COANE ee CONE KOU ON CS 


S$ python test_iterator.py 


aaa 
bbb 
Cee 
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4 
4. 
aaa 
bbb 
ee 
reached end of sequence 


An instance of a class which implements the __iter__ method, returning an iterator, is 
iterable. For example, it can be used in a for statement or in a list comprehension, or in 
a generator expression, or as an argument to the iter () built-in method. But, notice 
that the class most likely implements a generator method which can be called directly. 


Examples -- The following code implements an iterator that produces all the objects in a 
tree of objects: 


class Node: 
def __ init__(self, data, children=None) : 
self.initlevel = 0 
self.data = data 
if children is None: 


self.children = [] 
else: 
self.children = children 
GQek See anrtileveill (seit, anmtikeved) ) (seit vinit Level = ant Lever 
def get_initlevel (self): return self.initlevel 
oleic ciclelolasliel (seulie,,  (elavil ilxel)) 


self.children.append (child) 
def get_data(self): 
return self.data 
def get_children(self): 
return self.children 
def show_tree(self, level): 
self.show_level (level) 
print 'data: %s' % (self.data, ) 
Pom icine) aime Seder. clades ins 
child.show_tree(level + 1) 
def show_level (self, level): 
joiealinne, Y Ue ellewelyy 


# 
# Generator method #1 
# This generator turns instances of this class into iterable 


objects. 
# 
def walk_tree(self, level): 
yield (level, self, ) 
fom child ian seli.getochildren()i: 


for levell, treel in child.walk_tree(leveltl1): 

yield levell, treel 
def __iter (self): 
return self.walk_tree(self.initlevel) 
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Generator method #2 


3 
z 
# This generator uses a support function (walk_list) which calls 
fi this function to recursively walk the tree. 
# If effect, this iterates over the support function, which 
7 iterates over this function. 
z 
def walk_tree(tree, level): 
yield (level, tree) 
for child in walk_list (tree.get_children(), level+t1): 
yield child 
def walk_list (trees, level): 
for tree in trees: 
for tree in walk_tree(tree, level): 
yield tree 
# 
# Generator method #3 
# This generator is like method #2, but calls itself (as an 
iterator), 
# rather than calling a support function. 
# 
def walk_tree_recur(tree, level): 
yield (level, tree,) 
for child in tree.get_children(): 
woe lkesweubil \eieteeuil ahiak Wiyenllic aeieerer eerie (ela ike! ihkesweilail)) 4 


yield (levell, treel, ) 

def show_level (level): 

pring |! fos tee ll 
def test(): 

a7 = Node('777"') 

a6 = Node('666') 

a5 = Node('555') 

a4 = Node ('444') 

a3 = Node('333', [a4, a5]) 

az — Nede(t 22257) ac eau |) 

al = Node('1l11', [a2, a3]) 

initLevel = 2 

al.show_tree(initLevel) 

print '=' * 40 

fom evel, item am walk treei(alk, 


show_level (level) 
print 'item:', item.get_data() 
print '=' * 40 
for level, 
show_level (level) 
print 'item:', item.get_data () 


item in walk_tree_recur (al, 


initLevel) : 


initLevel): 
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print '=' * 40 
al.set_initlevel (initLevel) 
Ror skewela slteem slein, rellee 
show_level (level) 
print 'item:', item.get_data () 
iterl = iter(al) 
print iterl 
print iterl.next () 
Prime Atel ones () 
print iterl.next () 
print iterl.next () 
print iterl.next () 
() 
() 
ba 


print iterl.next 
print iterl.next 

## print iterl.ne 
meturn all 


siesta name == ' main ee 
test () 


Notes: 


e Aninstance of class Node is "iterable". It can be used directly in a for 
statement, a list comprehension, etc. So, for example, when an instance of Node 
is used in a for statement, it produces an iterator. 

e We could also call the Node.walk_method directly to obtain an iterator. 

e Method Node.walk_tree and functions walk_tree and 
walk_tree_recur are generators. When called, they return an iterator. They 
do this because they each contain a yield statement. 

e These methods/functions are recursive. They call themselves. Since they are 
generators, they must call themselves in a context that uses an iterator, for 
example in a for statement. 


1.7.4 Modules 


A module is a Python source code file. 


A module can be imported. When imported, the module is evaluated, and a module object 
is created. The module object has attributes. The following attributes are of special 
interest: 


e __doc__ -- The doc string of the module. 
e __name__ -- The name of the module when the module is imported, but the 
string "__main__" when the module is executed. 
e Other names that are created (bound) in the module. 
A module can be run. 


To make a module both import-able and run-able, use the following idiom (at the end of 
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the module): 


Ger amas ()\ie 
fe) 
fe) 
© 


LAE name == main 
main () 


Where Python looks for modules: 


e See sys.path. 

e Standard places. 

e Environment variable PYTHONPATH. 
Notes about modules and objects: 


e A module is an object. 

e A module (object) can be shared. 

e A specific module is imported only once in a single run. This means that a single 
module object is shared by all the modules that import it. 


1.7.4.1. Doc strings for modules 


Add docstrings as a triple-quoted string at or near the top of the file. See epydoc for a 
suggested format. 


1.7.5 Packages 
A package is a directory on the file system which contains a file named __init__.py. 
The __ init__.py file: 


e Why is it there? -- It makes modules in the directory "import-able". 

e Can__init__.py be empty? -- Yes. Or, just include a comment. 

e When is it evaluated? -- It is evaluated the first time that an application imports 
anything from that directory/package. 

e What can you do with it? -- Any code that should be executed exactly once and 
during import. For example: 

o Perform initialization needed by the package. 

o Make variables, functions, classes, etc available. For example, when the 
package is imported rather than modules in the package. You can also expose 
objects defined in modules contained in the package. 

e Define a variable named __al1___ to specify the list of names that will be 
imported by from my_package import ~*. For example, if the following is 
present in my_package/__init__.py: 
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Selle a eared eee 2a | 


Then, from my_package import * will import funcl and func2, but 
not other names defined in my_package. 
Note that___al1___ can be used at the module level, as well as at the package 
level. 
For more information, see the section on packages in the Python tutorial: 
http://docs.python.org/2/tutorial/modules.html#packages. 


Guidance and suggestions: 


e "Flat is better" -- Use the init__.py file to present a "flat" view of the API 
for your code. Enable your users to do import mypackage and then 
reference: 

o mypackage.iteml 

o mypackage.item2 

o mypackage.item3 

o Ete. 

Where item1, item2, etc compose the API you want your users to use, even 
though the implementation of these items may be buried deep in your code. 

e Use the init__.py module to present a "clean" API. Present only the items 
that you intend your users to use, and by doing so, "hide" items you do not intend 
them to use. 


1.8 Classes 


Classes model the behavior of objects in the "real" world. Methods implement the 
behaviors of these types of objects. Member variables hold (current) state. Classes enable 
us to implement new data types in Python. 


The class: statement is used to define a class. The class: statement creates a class 
object and binds it to a name. 


1.8.1 A simple class 


dean (OAS eikasics Au 
NN eteraaese pass 


To define a new style class (recommended), inherit from object or from another class 
that does. Example: 


In (20)) cellass A(object): 
ee pass 


Page 69 


A Python Book 


iia [22] s 

In [22]: a = A() 

Time 23) sa 

Owe |2ZS)|s — Mean A Clojecic ere WesxtsZicloiccre > 


1.8.2 Defining methods 


A method is a function defined in class scope and with first parameter self: 


in Ge | telkasis| Bob gecu)e 
esioeuais def show(self): 
wetienad Prime nei Teron Be 
Isy PAROS lo; 1B 0) 
In [108]: b.show() 
hello from B 


A method as we describe it here is more properly called an instance method, in order to 
distinguish it from class methods and static methods. 


1.8.3 The constructor 
The constructor is a method named init__. 


Exercise: Define a class with a member variable name and a show method. Use print 
to show the name. Solution: 


ime [PUOO | ss ellass: Aiob ject: 
Tynan cas Get — aimat (seis, mame): 
aentviesatat self.name = name 
Beavis e : def show(self): 
sdneeene print 'name: "%s"' % self.name 
In [111]: a = A('dave') 
iia | LP) 8 ale Slaton (() 
name: "dave" 


Notes: 
e The self variable is explicit. It references the current object, that is the object 
whose method is currently executing. 
e The spelling ("self") is optional, but everyone spells it that way. 
1.8.4 Member variables 


Defining member variables -- Member variables are created with assignment. Example: 


class A(object): 
Get 2 inti (sel, smame)i: 
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self.name = name 


A small gotcha -- Do this: 


In [28]: class A(object): 
mmauey eu def __ init__(self, items=None): 
swe if items is None: 
sreeeiene self.items = [] 
ebiets else: 
Shs viens self.items = items 


ial |LAShi| 2 wullsisiss 1335 

Sehr Cet Sanit (self, seems —|i) # wrong. list ctor 
evaluated only once. 

Sitserrs self.items = items 


In the second example, the def statement and the list constructor are evaluated only 
once. Therefore, the item member variable of all instances of class B, will share the same 
value, which is most likely not what you want. 


1.8.5 Calling methods 


e Use the instance and the dot operator. 
e Calling a method defined in the same class or a superclass: 
o From outside the class -- Use the instance: 


some_object.some_method () 
an_array_of_of_objects[1].another_method() 


o From within the same class -- Use self: 


self.a_method() 


o From with a subclass when the method is in the superclass and there is a 
method with the same name in the current class -- Use the class (name) or use 


super: 
SomesuperClass.__init (self, argl, arg2) 
super (CurrentClass, 
self) ont (angi, anrg2) 


e Calling a method defined in a specific superclass -- Use the class (name). 


1.8.6 Adding inheritance 


Referencing superclasses -- Use the built-in super or the explicit name of the 
superclass. Use of super is preferred. For example: 


dial |[LSiShis culevsiey IBV) 6 
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Naeayene Cer aainite = (Selgiys mame, SsaeZeiae 
Sheers super (B, self) .__init__ (name) 
aerate # A. init__(self, name) # an older alternative 


Scseenens self.size = size 


The use of super () may solve problems searching for the base class when using 
multiple inheritance. A better solution is to not use multiple inheritance. 


You can also use multiple inheritance. But, it can cause confusion. For example, in the 
following, class C inherits from both A and B: 


class CA; Bi: 


Python searches superclasses MRO (method resolution order). If only single inheritance 
is involved, there is little confusion. If multiple inheritance is being used, the search order 
of super classes can get complex -- see here: 
http://www.python.org/download/releases/2.3/mro 


For more information on inheritance, see the tutorial in the standard Python 
documentation set: 9.5 Inheritance and 9.5.1 Multiple Inheritance. 


Watch out for problems with inheriting from multiple classes that have a common base 
class. 


1.8.7 Class variables 


e Also called static data. 
e Aclass variable is shared by instances of the class. 
e Define at class level with assignment. Example: 


class A(object): 
size = 5 
def get_size(self): 
return A.size 


e Reference with classname.variable. 
e Caution: self.variable = xcreates anew member variable. 


1.8.8 Class methods and static methods 


Instance (plain) methods: 


e An instance method receives the instance as its first argument. 
Class methods: 


e Aclass method receives the class as its first argument. 
e Define class methods with built-in function classmethod() or with decorator 
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@classmethod. 
e See the description of classmethod() built-in function at "Built-in 
Functions": http://docs.python.org/2/library/functions.html#classmethod 
Static methods: 


e A static method receives neither the instance nor the class as its first argument. 
e Define static methods with built-in function staticmethod() or with 
decorator @staticmethod. 
e See the description of staticmethod() built-in function at "Built-in 
Functions": http://docs.python.org/2/library/functions.html#staticmethod 
Notes on decorators: 


e A decorator of the form @afunc is the same asm = afunc(m). So, this: 


@afunc 
def m(self): pass 


is the same as: 


def m(self): pass 
m = afunc(m) 


e Youcan use decorators @classmethod and @staticmethod (instead of the 
classmethod() and staticmethod () built-in functions) to declare class 
methods and static methods. 

Example: 


class B(object): 
Count = 0 


def dup_string(x): 
si = “ssias os 9 
isiSewheia fell 

dup_string = staticmethod(dup_string) 


x, x, ) 


@classmethod 
def show_count (cls, msg): 


° 


prime “ss “adi “2 msg, cls. Count, ) 


def test(): 
joneakiane 1s}, (6lblos sere abiave) (i eilexerel)) 
B.show_count ('here is the count: ') 


An alternative way to implement "static methods" -- Use a "plain", module-level 
function. For example: 


dha (abs lesz “Stine evonbiialia (()) = 
: A.count += 1 


IE iat coe 
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ine [iis der sdecucount (): 
: A.count -= 1 
alistaeenlieSsllies 
inal Se seilkarstcy Avs 
5 count = 0 


def get_count (self): 
Ste wuain AV, COibuae 


In [4 

In [4 a = A() 

lia LS a.get_count () 
Owe LS 0 

lia | 6 

Ia (16 

irae (56 ImeEucouneE () 
aTSsteveam [ie abighe} exoyblione (()) 
ibiae fee) a.get_count () 
@ythe es 2 

iba |S) 

In [9]: b = A() 

ine (Oli begeeacounte() 
ele Ali) sg 2 


1.8.9 Properties 


The property built-in function enables us to write classes in a way that does not require a 
user of the class to use getters and setters. Example: 


class TestProperty (object): 
def __init__ (self, description): 
self._description = description 
def _set_description(self, description): 
print 'setting description' 
self._description = description 
def _get_description(self): 
print 'getting description' 
return self._description 
description = property (_get_description, _set_description) 


The property built-in function is also a decorator. So, the following is equivalent to 
the above example: 


class TestProperty (object): 
def __init__(self, description): 
self._description = description 


@property 

def description(self): 
print 'getting description' 
return self._description 
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@description.setter 

def description(self, description): 
print 'setting description' 
self._description = description 


Notes: 


e We mark the instance variable as private by prefixing it with and underscore. 
e The name of the instance variable and the name of the property must be different. 
If they are not, we get recursion and an error. 
For more information on properties, see Built-in Functions -- properties -- 
http://docs.python.org/2/library/functions.html#property 


1.8.10 Interfaces 


In Python, to implement an interface is to implement a method with a specific name and a 
specific arguments. 


"Duck typing" -- If it walks like a duck and quacks like a duck ... 


One way to define an "interface" is to define a class containing methods that have a 
header and a doc string but no implementation. 


Additional notes on interfaces: 


e Interfaces are not enforced. 
e Aclass does not have to implement all of an interface. 


1.8.11 New-style classes 


A new-style class is one that subclasses ob ject or a class that subclasses ob ject (that 
is, another new-style class). 


You can subclass Python's built-in data-types. 


e A simple example -- the following class extends the list data-type: 


eilkereisy 16. (Cilakene) 6 
def get_len(self): 
return len(self) 


OS Cl 2 SiS) 
c.get_len () 


Cr CCUM 227 837447557 60, 1, oo) 
print c.get_len() 
# Piet nts MMs 


e A slightly more complex example -- the following class extends the dictionary 
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data-type: 


ellvaiess ID (Cluliei=)) 6 
def __ init__(self, data=None, name='no_name'): 
if data is None: 
data = {} 
Clivete,  aliguiie — (@sieultc > Yolehec!)) 
self.name = name 
def get_len(self): 
return len(self) 
def get_keys (self): 
content = [] 
£or key iin ‘sede: 
content.append (key) 
contentstr — %, ‘. join (content) 
return contentstr 
def get_name (self): 
return self.name 


def test(): 
a= Dit aat =) ahi Ubbk 2A Vee Sas) 
7 Prints, "3 
print d.get_len() 
+ Peiinises Vata «Cee solo” & 
print d.get_keys () 
# Prints "no_name" 
print d.get_name () 


Some things to remember about new-style classes: 


e Inorder to be new-style, a class must inherit (directly or indirectly) from 
object. Note that if you inherit from a built-in type, you get this automatically. 
e New-style classes unify types and classes. 


e Youcan subclass (built-in) types such as dict, str, list, file, etc. 

e The built-in types now provide factory functions: dict (), str(), int (), 
file(),etc. 

e The built-in types are introspect-able -- Use x.__class_, 
clit (x, CGlass__)§isimstance (x, Lisi) Meter 


e New-style classes give you properties and descriptors. 
e New-style classes enable you to define static methods. Actually, all classes enable 
you to do this. 
e Anew-style class is a user-defined type. For an instance of a new-style class x, 
type (x) is the same as x.__class__. 
For more on new-style classes, see: http://www.python.org/doc/newstyle/ 


Exercises: 


e Write aclass and a subclass of this class. 
o Give the superclass one member variable, a name, which can be entered when 
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an instance is constructed. 

o Give the subclass one member variable, a description; the subclass constructor 
should allow entry of both name and description. 

o Puta show() method in the superclass and override the show () method in 
the subclass. 


Solution: 
class A(object): 
dak Sinaia (Seis tmame) = 
self.name = name 


def show(self): 
print 'name: %s' % (self.name, ) 


class B(A): 
death inate (Selb ee Mame; sncde sie) a 
Aye inv (sedit, name) 
self.desc = desc 
def show(self): 
A.show (self) 
print ‘desc: @s' % (self.desc, ) 


1.8.12 Doc strings for classes 


Add docstrings as a (triple-quoted) string beginning with the first line of a class. See 
epydoc for a suggested format. 


1.8.13 Private members 


Add an leading underscore to a member name (method or data variable) to "suggest" that 
the member is private. 


1.9 Special Tasks 


1.9.1 Debugging tools 
pdb -- The Python debugger: 


e Start the debugger by running an expression: 


pdb.run('expression') 


Example: 


aie name == '  main__': 
LMP Oneiss pes) 


pabe wun (madi ()") 


e Start up the debugger at a specific location with the following: 
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import pdb; pdb.set_trace() 


Example: 
alae name == '  main__': 
import pdb 
jreloys See esecierS (()) 
main () 


e Get help from within the debugger. For example: 


(Pdb) help 
(Pdb) help next 


Can also embed [Python into your code. See 
http://ipython.scipy.org/doc/manual/manual.html. 


ipdb -- Also consider using ipdb (and IPython). The ipdb debugger interactive 
prompt has some additional features, for example, it does tab name completion. 


Inspecting: 


e import inspect 

e See http://docs.python.org/lib/module-inspect.html. 

e Don't forget to try dir (obj) and type (obj) and help (ob}), first. 
Miscellaneous tools: 


e id(obj) 

e globals() andlocals(). 

e dir(obj) -- Returns interesting names, but list is not necessarily complete. 

e obj.__class__ 

e cls.__bases__ 

e obj.__class__.__bases__ 

e obj.__doc_. But usually, help (obj) is better. It produces the doc string. 

e Customize the representation of your class. Define the following methods in your 
class: 
Oo __repr___() -- Called by (1) repr (), (2) interactive interpreter when 


representation is needed. 
fo) str__() -- Called by (1) str (), (2) string formatting. 


pdb is implemented with the cmd module in the Python standard library. You can 
implement similar command line interfaces by using cmd. See: cmd -- Support for 
line-oriented command interpreters -- http://docs.python.org/lib/module-cmd.html. 


1.9.2 File input and output 
Create a file object. Use open (). 


This example reads and prints each line of a file: 
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def test(): 
fe ile G tmp coyy;, oie ')) 
ioe Ibias shia ie8 
jouealians  lkakiareso U7) llaviatey, teais\cae ake) (9) 
£f.close() 


test () 


Notes: 


e A text file is an iterable. It iterates over the lines in a file. The following is a 


common idiom: 


dimes 1) = file(filename, 'r') 
fone) lei sera erates 
process_a_line (line) 
infile.close() 


e string.rstrip() strips new-line and other whitespace from the right side of 
each line. To strip new-lines only, but not other whitespace, try rstrip('\n'). 
e Other ways of reading from a file/stream object: my_file.read(), 


my_file.readline(),my_file.readlines(), 
This example writes lines of text to a file: 


def test(): 
£ = file(*tmp-cext';, ‘w') 
Eom eh in Yabecdero : 
owe techn + 10) 
se Spe aes) (2 Nia) 
£.close() 


test () 


Notes: 


e The write method, unlike the print statement, does not automatically add 


new-line characters. 


e Must close file in order to flush output. Or, use my_file.flush(). 
And, don't forget the with: statement. It makes closing files automatic. The following 
example converts all the vowels in an input file to upper case and writes the converted 


lines to an output file: 


SHINOICHEIS JSGIASLING 


seoue ILshine: shiny sunliesL Ike < 
line = line.rstrip() 


def show_file(infilename, outfilename): 

tran_table = string.maketrans('aeiou', 'AEIOU') 

with open(infilename, 'r') as infile, open(outfilename, 
outfile: 


CUbmi Ne write (Ss Wnts slimes tramsikate(t cammtalolle))) 


Page 79 


A Python Book 


1.9.3 Unit tests 


For more documentation on the unit test framework, see unittest -- Unit testing 
framework -- http://docs.python.org/2/library/unittest.html#module-unittest 


For help and more information do the following at the Python interactive prompt: 


SSS sbulsiougie Uline 
>>> help (unittest) 


And, you can read the source: Lib/unittest .py in the Python standard library. 


1.9.3.1 A simple example 


Here is a very simple example. You can find more information about this primitive way 
of structuring unit tests in the library documentation for the unittest module Basic 
example -- http://docs.python.org/lib/minimal-example.html 


import unittest 
class UnitTests02 (unittest.TestCase): 


def testFoo(self): 
self.failUnless (False) 


class UnitTests01(unittest.TestCase): 


def testBar0l(self): 
self.failUnless (False) 


def testBar02 (self): 
self.failUnless (False) 


Ges amaciene (sine 
unittest.main() 


SLE name == main 
main () 


Notes: 


e Thecall to unittest.main() runs all tests in all test fixtures in the module. It 
actually creates an instance of class Test Program in module 
Lib/unittest.py, which automatically runs tests. 

e Test fixtures are classes that inherit from unittest .TestCase. 

e Within a test fixture (a class), the tests are any methods whose names begin with 
the prefix "test". 

e Inany test, we check for success or failure with inherited methods such as 
faillf(), failUnless(), assertNotEqual (), etc. For more on these 
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methods, see the library documentation for the unittest module TestCase 
Objects -- http://docs.python.org/lib/testcase-objects.html. 

e If you want to change (1) the test method prefix or (2) the function used to sort 
(the order of) execution of tests within a test fixture, then you can create your own 
instance of class unittest .Test Loader and customize it. For example: 


def main(): 
my_test_loader = unittest.TestLoader () 
my_test_loader.testMethodPrefix = 'check' 
my_test_loader.sortTestMethodsUsing = my_cmp_func 


unittest.main(testLoader=my_test_loader) 


alls name == '  main__': 


But, see the notes in section Additional unittest features for instructions on a 
(possibly) better way to do this. 


1.9.3.2 Unit test suites 


Here is another, not quite so simple, example: 


#!/usr/bin/env python 
import sys, popen2 
import getopt 

import unittest 


class GenTest (unittest.TestCase): 


def test_l_generate(self): 


emd = "python ../generateDS.py -f -o out2sup-.py -s out2sub.py 
people.xsd' 

outfile, infile = popen2.popen2 (cmd) 

result = outfile.read() 

outfile.close() 


infile.close() 
self.failUnless(len(result) == 0) 


def test_2_compare_superclasses (self): 

nel = Viclkbiew Wlvieilsulo joy, Cue Sule coy! 
outfile, infile = popen2.popen2 (cmd) 
outfile, infile = popen2.popen2 (cmd) 
result = outfile.read() 
outfile.close() 
infile.close() 
#print 'len(result):', len(result) 
# Ignore the differing lines containing the date/time. 
#self.failUnless(len(result) < 130 and 

result.find('Generated') > -1) 
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self.failUnless (check_result (result) ) 


def test_3_compare_subclasses (self): 

chnel = Vichtiew bic ilsule joy OWlie Zullo). joy! 
outfile, infile = popen2.popen2 (cmd) 
outfile, infile = popen2.popen2 (cmd) 
result = outfile.read() 
outfile.close() 
infile.close() 
# Ignore the differing lines containing the date/time. 
#self.failUnless(len(result) < 130 and 

result.find('Generated') > -1) 
self.failUnless (check_result (result) ) 


def check_result (result): 

flagl = 0 

flag2 = 0 

lines = result.split('\n"') 

lenl = len(lines) 

tf end <= 5: 
tedkere il = I 

Sle Nia ee esea(ime si: )) 

it isle hime(vGeneirated" ) >) i: 
flag2 = 1 


return flagl and flag2 


# Make the test suite. 
def suite(): 
# The following is obsolete. See Lib/unittest.py. 
#return unittest.makeSuite (GenTest) 
loader = unittest.TestLoader () 
# or alternatively 
# loader = unittest.defaultTestLoader 
testsuite = loader.loadTestsFromTestCase (GenTest) 
return testsuite 


# Make the test suite and run the tests. 

def test(): 
testsuite = suite() 
runner = unittest.TextTestRunner(sys.stdout, verbosity=2) 
runner.run(testsuite) 


USAGE_TEXT = """ 
Usiage: 
python test.py [options] 
Options: 
=, ——nieyiho Display this help message. 
Example: 


python test.py 
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ww 


def 


def 


aLie 


usage (): 
print USAGE_TEXT 


sys.exit (-1) 


main(): 
args = sys.argv[1:] 
dealer 
opts, args = getopt.getopt(args, ‘'h', ['help']) 
SESS € 
usage () 
relink = 1 
igGne jeiey welll shar eysiets)s 
ie Oe seme (SY ——teiliiom) z 
usage () 
if len(args) != 
usage () 
esis”) 


(6) 


name == ' main We 


main() 
#import pdb 
#pdb.run('main() ') 


1.9.3.3 


GenTest is our test suite class. It inherits from unittest .TestCase. 
Each method in GenTest whose name begins with "test" will be run as a test. 
The tests are run in alphabetic order by method name. 

Defaults in class Test Loader for the test name prefix and sort comparison 
function can be overridden. See 5.3.8 TestLoader Objects -- 
http://docs.python.org/lib/testloader-objects.html. 

A test case class may also implement methods named setUp () and 
tearDown () to be run before and after tests. See 5.3.5 TestCase Objects -- 
http://docs.python.org/lib/testcase-objects.html. Actually, the first test method in 
our example should, perhaps, be a set Up () method. 

The tests use calls such as sel £.failUnless () to report errors. These are 
inherited from class Test Case. See 5.3.5 TestCase Objects -- 
http://docs.python.org/lib/testcase-objects.html. 

Function suite () creates an instance of the test suite. 

Function test () runs the tests. 


Additional unittest features 


And, the following example shows several additional features. See the notes that follow 
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def 


def 


class UnitTe 


def test 
self 


import unittest 


sts02 (unittest.TestCase): 
Foo(self): 
.failUnless (False) 


def chec 
self 


kBarOl (self): 
.failUnless (False) 


class UnitTe 


# Note 1 
def setU 
joie aia 
def tear 
joue@aL ial 
def test 
joueaLia 
self 


sts0l1(unittest.TestCase): 


p(self): 

@ “setting Up Unrtelesesdi" 
Down (self): 

t ‘tearing down UnitTests01' 
Bar0l (self): 

t 'testing testBar0l1' 
.failUnless (False) 


def test 
joueaLial 
self 


Barz (sels): 
t 'testing testBar02' 
.failUnless (False) 


ie BOKGe sO jeevsic_ I iO) e 


name = ' 


EISISISNe ce ial 


compare_ 
if namel 
1 SEW 
elif nam 
Sie yul 
else: 
Letu 


make_sui 


mona' 
ot name.startswith('mo') 


names (namel, name2): 
< name2: 

iia IL 

el > name2: 

win JL 


isin (6) 


te(): 


def 


SLae 


suite = 
# Note 2 
suite.ad 


# Note 3 
suite.ad 
# Note 4 


unittest.TestSuite() 


dTest (unittest.makeSuite (UnitTests0l1, 


sortUsing=compare_names) ) 


dTest (unittest .makeSuite (UnitTests02, 


prefix='check') ) 


suite.addTest (unittest.FunctionTestCase (function_test_1) ) 


return s 


tmleiitial (()) & 
suite = 
runner = 
is \Uhalighenie ge 


name 


uite 


make_suite() 
unittest.TextTestRunner () 
un (suite) 


== main vhs 
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main () 


Notes: 


1. If you run this code, you will notice that the setUp and tearDown methods in 
class UnitTests0O1 are run before and after each test in that class. 

2. Wecan control the order in which tests are run by passing a compare function to 
the makeSuite function. The default is the cmp built-in function. 

3. We can control which methods in a test fixture are selected to be run by passing 
the optional argument prefix to the makeSuite function. 

4. If we have an existing function that we want to "wrap" and run as a unit test, we 
can create a test case from a function with the Funct ionTestCase function. If 
we do that, notice that we use the assert statement to test and possibly cause 
failure. 


1.9.3.4 Guidance on Unit Testing 
Why should we use unit tests? Many reasons, including: 


e Without unit tests, corner cases may not be checked. This is especially important, 
since Python does relatively little compile time error checking. 

e Unit tests facilitate a frequent and short design and implement and release 
development cycle. See ONLamp.com -- Extreme Python -- 
http://www.onlamp.com/pub/a/python/2001/03/28/pythonnews.html and What is 
XP -- http://www.xprogramming.com/what_is_xp.htm. 

e Designing the tests before writing the code is "a good idea". 

Additional notes: 


e Ina test class, instance methods setUp and tearDown are run automatically 
before each and after each individual test. 

e Ina test class, class methods setUpClass and tearDownClass are run 
automatically once before and after all the tests in a class. 

e Module level functions setUpModule and tearDownModule are run before 
and after any tests in a module. 

e Insome cases you can also run tests directly from the command line. Do the 
following for help: 


S$ python -m unittest -—-help 


1.9.4 doctest 


For simple test harnesses, consider using doctest. With doctest you can (1) runa 
test at the Python interactive prompt, then (2) copy and paste that test into a doc string in 
your module, and then (3) run the tests automatically from within your module under 


Page 85 


A Python Book 


Cclo@cesick 


There are examples and explanation in the standard Python documentation: 5.2 doctest -- 
Test interactive Python examples -- http://docs.python.org/lib/module-doctest.html. 


A simple way to use doctest in your module: 


1. 


3. 


Here is 


Run several tests in the Python interactive interpreter. Note that because 
doctest looks for the interpreter's ">>>" prompt, you must use the standard 
interpreter, and not, for example, IPython. Also, make sure that you include a line 
with the ">>>" prompt after each set of results; this enables doctest to 
determine the extent of the test results. 

Use copy and paste, to insert the tests and their results from your interactive 
session into the docstrings. 

Add the following code at the bottom of your module: 


def _test(): 
import doctest 
doctest.testmod () 
alge name == "— main __": 
_test () 


an example: 


def 


def 


abe 


12 (Gal) 


ww 


Print something funny. 


Soe (Ul) 
10 
SS (2) 
—10 
>> (3) 
0 
wo 
if n == 
Bet Urea 1G 
elif n == 
reetesblisial “= J16) 
else: 
return 0 


test (): 
import doctest, test_doctest 
doctest.testmod(test_doctest) 


name == main ree 


test () 


And, here is the output from running the above test with the —v flag: 
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S python test_doctest.py -v 
Running test_doctest.__doc__ 

0 of 0 examples failed in test_doctest.__doc 
Running test_doctest.f.__doc__ 
Wieyyabioweg se (Al) 

Expecting: 10 

ok 

Aeabiqve; Be ac (02) 

Expecting: -10 

ok 

Wiewabiowps 22 ((3)) 

Expecting: 0 


ok 

0 of 3 examples failed in test_doctest.f.__doc__ 
Running test_doctest.test.__doc__ 

0 of 0 examples failed in test_doctest.test.__doc__ 


2 items had no tests: 
test_doctest 
test_doctest.test 

1 items passed all tests: 

aS cests: an testacdoctest. 

3 tests ans aeemsi- 

3 passed and 0 failed. 

Test passed. 


1.9.5 The Python database API 


Python database API defines a standard interface for access to a relational database. 


In order to use this API you must install the database adapter (interface module) for your 
particular database, e.g. PostgreSQL, MySQL, Oracle, etc. 


You can learn more about the Python DB-API here: 
http://www.python.org/dev/peps/pep-0249/ 


The following simple example uses sqlite3 -- http://docs.python.org/2/library/sqlite3.html 


#!/usr/bin/env python 


wo 
Create a relational database and a table in it. 


Add some records. 
Read and display the records. 


ww 


siijeionas, Shs 
import sqlite3 


def create_table(db_name): 


con = sqlite3.connect (db_name) 
Culecom = Com. cumesoie() 
cursor.execute('''CREATE TABLE plants 
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(deme ext, Gesie tesa, Sale alitite) i) 
cursor.execute ( 


"TENSERG INO plants VALUES ('tomato'7 “red and quucy 7 
dD Ve ae aS 
(ohbligteioue 5 (Sedeve bliciS (( 
INSERT TNLO planes VALUES (pepper. |) TGneem and ieruneiny  ; 
Py aE 
Gliese se >seeiblie'S (VY VION SIR INN) jollenaies: WAINUIES, ( eoeyojersie,  Vjeluleiebes! 
Pry ES) 


con.commit () 
con.close() 


def retrieve (db_name) : 


con = sqlite3.connect (db_name) 

CUBSOr — ICoOn cuts om ()) 
cursor.execute('''select * from plants''') 
rows = cursor.fetchall () 

print rows 

piting t=) = 40) 

Cunmson.execube ("! “select ~ from plants? *™) 


Om SOW in Cliasi@is.: 
Deine Bow 
con.close() 


Glaie ‘cise ()) 
ands: = Svis.anom [a] 
if len(args) != 1: 


sys.stderr.write('\nusage: test_db.py <db_name>\n\n') 
sys.exit (1) 

db_name = args[0] 

create_table (db_name) 

retrieve (db_name) 


test () 


1.9.6 Installing Python packages 
Simple: 


S joel sieiewisicjexy lowell il 
S python setup.py install # as root 


More complex: 


e Look for a README or INSTALL file at the root of the package. 
e Type the following for help: 


S python setup.py cmd ——help 
S python setup.py -——-help-commands 
S fovclaci Secu =—laeulicy [emolll mele 3 4. || 


e And, for even more details, see Installing Python Modules -- 
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http://docs.python.org/inst/inst.html 
pip is becoming popular for installing and managing Python packages. See: 
https://pypi.python.org/pypi/pip 
Also, consider using virtualenv, especially if you suspect or worry that installing 
some new package will alter the behavior of a package currently installed on your 
machine. See: https://pypi.python.org/pypi/virtualenv. virtualenv creates a directory 
and sets up a Python environment into which you can install and use Python packages 
without changing your usual Python installation. 


1.10 More Python Features and Exercises 


[As time permits, explain more features and do more exercises as requested by class 
members. | 


Thanks to David Goodger for the following list or references. His "Code Like a 
Pythonista: Idiomatic Python" 
(http://python.net/~goodger/projects/pycon/2007/idiomatic/) is worth a careful reading: 


e "Python Objects", Fredrik Lundh, http://www.effbot.org/zone/python-objects.htm 

e "How to think like a Pythonista", Mark Hammond, 
http://python.net/crew/mwh/hacks/objectthink. html 

e "Python main() functions", Guido van Rossum, 
http://www.artima.com/weblogs/viewpost.jsp?thread=4829 

e "Python Idioms and Efficiency", http://jaynes.colorado.edu/PythonIdioms.html 

e "Python track: python idioms", 
http://www.cs.caltech.edu/courses/cs 1 1/material/python/misc/python_idioms.html 

e "Be Pythonic", Shalabh Chaturvedi, http://shalabh.infogami.com/Be_Pythonic2 

e "Python Is Not Java", Phillip J. Eby, 
http://dirtsimple.org/2004/12/python-is-not-java.html 

e "What is Pythonic?", Martijn Faassen, 
http://faassen.n--tree.net/blog/view/weblog/2005/08/06/0 

e "Sorting Mini- HOWTO", Andrew Dalke, 
http://wiki.python.org/moin/HowTo/Sorting 

e "Python Idioms", http://www.gungfu.de/facts/wiki/Main/PythonIdioms 

e "Python FAQs", http://www.python.org/doc/faq/ 
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2 Part 2 -- Advanced Python 


2.1 Introduction -- Python 201 -- (Slightly) Advanced Python Topics 


This document is intended as notes for a course on (slightly) advanced Python topics. 


2.2 Regular Expressions 


For more help on regular expressions, see: 


re - Regular expression operations http://docs.python.org/library/re.html 
Regular Expression HOWTO -- http://docs.python.org/howto/regex.html 


2.2.1 Defining regular expressions 


A regular expression pattern is a sequence of characters that will match sequences of 
characters in a target. 


The patterns or regular expressions can be defined as follows: 


Literal characters must match exactly. For example, "a" matches "a". 

Concatenated patterns match concatenated targets. For example, "ab" ("a" 

followed by "b") matches "ab". 

Alternate patterns (separated by a vertical bar) match either of the alternative 

patterns. For example, "(aaa)l(bbb)" will match either "aaa" or "bbb". 

Repeating and optional items: 

o "abc*" matches "ab" followed by zero or more occurances of "c", for example, 
"ab", "abc", "abcc", etc. 

o "abc+" matches "ab" followed by one or more occurances of "c", for example, 
"abc", "abcc", etc, but not "ab". 

o "abc?" matches "ab" followed by zero or one occurances of "c", for example, 
"ab" or "abc". 

Sets of characters -- Characters and sequences of characters in square brackets 

form a set; a set matches any character in the set or range. For example, "[abc]" 

matches "a" or "b" or "c". And, for example, "[_a-z0-9]" matches an underscore 

or any lower-case letter or any digit. 

Groups -- Parentheses indicate a group with a pattern. For example, "ab(cd)*ef" is 

a pattern that matches "ab" followed by any number of occurances of "cd" 

followed by "ef", for example, "abef", "abcdef", "abcdcdef", etc. 

There are special names for some sets of characters, for example "\d" (any digit), 


mom 
: 
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"\w" (any alphanumeric character), "\W" (any non-alphanumeric character), etc. 

More more information, see Python Library Reference: Regular Expression 

Syntax -- http://docs.python.org/library/re.html#regular-expression-syntax 
Because of the use of backslashes in patterns, you are usually better off defining regular 
expressions with raw strings, e.g. r"abc". 


2.2.2 Compiling regular expressions 


When a regular expression is to be used more than once, you should consider compiling 
it. For example: 


import sys, re 


pat = re.compile('aa[bc] *dd"') 
while 1: 
line = raw inpur( hater a lime (Mo" two quate) 2") 
if line == 'q': 
break 


if pat.search(line): 

print 'matched:', line 
else: 

Diane nos Mateh= "lene 


Comments: 
e We import module re in order to use regular expresions. 
@e xre.compile() compiles a regular expression so that we can reuse the 
compiled regular expression without compiling it repeatedly. 
2.2.3 Using regular expressions 
Use match () to match at the beginning of a string (or not at all). 
Use search () to search a string and match the first string from the left. 


Here are some examples: 


>>> import re 

>>> pat = re.compile('aa[0-9]*bb') 

>>> x = pat.match('aal234bbccddee') 

SS> 

<_sre.SRE_Match object at 0x401e9608> 
>>> x = pat.match('xxxxaal234bbccddee' ) 
SS 

>>> type (x) 

<type 'NoneType'> 

>>> xX = pat.search('xxxxaal234bbccddee") 
SSS Se 
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<_sre.SRE_Match object at 0x401e9608> 


Notes: 


e When a match or search is successful, it returns a match object. When it fails, it 


returns None. 
e Youcan also call the corresponding functions match and search in the re module, 


e.g.: 


>>> x = re.search(pat, 'xxxxaal234bbccddee") 
SES 
<_sre.SRE_Match object at 0x401e9560> 


For a list of functions in the re module, see Module Contents -- 
http://docs.python.org/library/re.html#module-contents. 


2.2.4 Using match objects to extract a value 


Match objects enable you to extract matched sub-strings after performing a match. A 
match object is returned by successful match. The part of the target available in the match 
object is the portion matched by groups in the pattern, that is the portion of the pattern 
inside parentheses. For example: 


ine [Gol mer — see searen (is"nemoint = Ge )s wide mes (NGA) el ivenioine sl 23 
width: 456') 

im | 7Ols mo. Groups! () 

OHENL TACH Sy ke Sith Use) 


Here is another example: 


import sys, re 


Targets = [ 
"There are <<25>> sparrows.', 
"Rh see <<15>> finches. ' 
"There is nothing here.', 


] 


Glaie icissie ()) € 
joel) = ees Comosker((U << (0S y=!) 
for line in Targets: 
mo = pat.search (line) 
aLie MONE 
value = mo.group (1) 
print 'value: %s' % value 
elses 
pring Mme maze" 


test () 


When we run the above, it prints out the following: 
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weulluies 25 

wellwes 5 

no match 

Explanation: 

e Inthe regular expression, put parentheses around the portion of the regular 
expression that will match what you want to extract. Each pair of parentheses 
marks off a group. 

e After the search, check to determine if there was a successful match by checking 
for a matching object. "pat.search(line)" returns None if the search fails. 

e Ifyou specify more than one group in your regular expression (more that one pair 


of parentheses), then you can use "value = mo.group(N)" to extract the value 
matched by the Nth group from the matching object. "value = mo.group(1)" 
returns the first extracted value; "value = mo.group(2)" returns the second; etc. An 
argument of 0 returns the string matched by the entire regular expression. 


In addition, you can: 


Use "values = mo.groups()" to get a tuple containing the strings matched by all 
groups. 

Use "mo.expand()" to interpolate the group values into a string. For example, 
"mo.expand(r'value1: \l value2: \2')"inserts the values of the first and second 
group into a string. If the first group matched "aaa" and the second matched 
"bbb", then this example would produce "value1: aaa value2: bbb". For example: 


in iol mor = men searel (atin OVE wi KNGIA)T te Vins. 323 
w: 456') 

in i772 Mosexpand (re Heaghe = Vl) Wadi he 2) 

Ou lly file “Hemghies Zs) Waidibhr= 456. 


2.2.5 Extracting multiple items 


You can extract multiple items with a single search. Here is an example: 


import sys, re 


cle = WS ceomowle( cist (0 >2)I|)loley(( [Os ) ce” )) 


while 1: 
line = raw input 'enter a line ("oq" to guit):") 
if line == 'q': 
break 
mo = pat.search (line) 
i Mor 


valuel, value2 = mo.group(1, 

print 'valuel: %s value2: %s' % (valuel, value2) 
else: 

joxeaine, Viale) imene lay! 
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Comments: 


e Use multiple parenthesized substrings in the regular expression to indicate the 
portions (groups) to be extracted. 

e "mo.group(1, 2)" returns the values of the first and second group in the string 
matched. 

e We could also have used "mo.groups()" to obtain a tuple that contains both 
values. 

e Yet another alternative would have been to use the following: print 
Mowexpand(r'valuels Vil valae2: 9 \2")). 


2.2.6 Replacing multiple items 


A simple way to perform multiple replacements using a regular expression is to use the 
re.subn() function. Here is an example: 


in [sis ceastibmi(tse! Vote) Yaa ehere are 20s) bands sirkinc) im 2 
trees') 
Out fei: (ehere varcer** fords sate lng wim ~~ trees” 7) 2) 


For more complex replacements, use a function instead of a constant replacement string: 


import re 


def repl_func (mo): 
sl = mo.group (1) 
SZ ee Se he mnGsuls) 
return s2 


def test(): 
pat = r'(\dt)' 
in_str = 'there are 2034 birds in 21 trees' 
Obie Sei, Coble = we SUlom(joaic, well sewn, ail Scie) 
joneiong § Valnore  Weasyul er abinl isiene 
rim Vous Wass 3 seule ws tie 
Prime NMCOUunes ser o. Count 


test () 


And when we run the above, it produces: 


platalys Vohere are: 20S4 Voarredis) aim Ai emcees)” 
Ours “there vane “*** pies Tine +4 trees" 
G@@iuater ae 

Notes: 


e The replacement function receives one argument, a match object. 
e The re.subn() function returns a tuple containing two values: (1) the string 
after replacements and (2) the number of replacements performed. 
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Here is an even more complex example -- You can locate sub-strings (slices) of a match 
and replace them: 


import sys, re 


ele = wWeSseomowIle (Ysa (WS2)l|* Isle OSs )coy )) 
while 1: 
line = raw_input('Enter a line ("q" to quit): ') 
if line == 'q': 
break 
mo = pat.search (line) 
i mor 


valuel, value2 = mo.group(l1, 2) 
startl = mo.start(1l) 

endl = mo.end(1) 

StartzZ = mo.start (2) 

end2 = mo.end(2) 


pring 'valtici= Acs. “sSitvactils ce endl: sd" so i(valtiel, Vscartely, 
endl) 

pring '‘valwe2: ss" (Seanez: “sd jend2: 30" )s (valle stareZ, 
end2) 

repll = raw_input ('Enter replacement #1: ') 

repl2 = raw_input('Enter replacement #2: ') 

newline = (line[:start1l] + repll + line[endl:start2] + 


repl2 line[end2:]) 
print ‘newline: %s' % newline 
elses 
pLIne ne mateh” 


Explanation: 


e Alternatively, use "mo.span(1)" instead of "mo.start(1)" and "mo.end(1)" in order 
to get the start and end of a sub-match in a single operation. "mo.span(1)"returns a 
tuple: (start, end). 

e Put together a new string with string concatenation from pieces of the original 
string and replacement values. You can use string slices to get the sub-strings of 
the original string. In our case, the following gets the start of the string, adds the 
first replacement, adds the middle of the original string, adds the second 
replacement, and finally, adds the last part of the original string: 


newline = line[:start1l] + repll + line[endl:start2] + 
repl2 + line[end2:] 


You can also use the sub function or method to do substitutions. Here is an example: 


import sys, re 


pat = re.compile('[0-9]+") 


print 'Replacing decimal digits.' 
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while 1: 
target = raw_input('Enter a target line ("q" to quit): ') 
Lt target == “q's 
break 
repl = raw_input('Enter a replacement: ') 
result = pat.sub(repl, target) 
brine Trestles “cst o meswilit 


Here is another example of the use of a function to insert calculated replacements. 


import sys, re, string 
pat = re.compile('[a-m]+') 


def replacer (mo): 
return string.upper(mo.group (0) ) 


Pring !Upper—casang “ame 


while 1: 
target = raw_input('Enter a target line ("gq" to quit): ') 
if target == 'q': 
break 


result = pat.sub(replacer, target) 
pene Meeswiltts ss" so result 


Notes: 


e Ifthe replacement argument to sub is a function, that function must take one 
argument, a match object, and must return the modified (or replacement) value. 
The matched sub-string will be replaced by the value returned by this function. 
e Inour case, the function replacer converts the matched value to upper case. 
This is also a convenient use for a lambda instead of a named function, for example: 


import sys, re, string 
pat = re.compile('[a-m]+') 


pLine. YUpper— Casing. dm 


while 1: 
Garnget — raw input ("Enter a target lime (“q" to quae) ) 
if target == 'q': 
break 
result = pat.sub( 
lambda mo: string.upper(mo.group(0)), 
target) 


peime Mcesullte sss <5 mesult 


2.3 Iterator Objects 


Note 1: You will need a sufficiently recent version of Python in order to use iterators and 
generators. I believe that they were introduced in Python 2.2. 
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Note 2: The iterator protocol has changed slightly in Python version 3.0. 


Goals for this section: 


Learn how to implement a generator function, that is, a function which, when 
called, returns an iterator. 

Learn how to implement a class containing a generator method, that is, a method 
which, when called, returns an iterator. 

Learn the iterator protocol, specifically what methods an iterator must support and 
what those methods must do. 

Learn how to implement an iterator class, that is, a class whose instances are 
iterator objects. 

Learn how to implement recursive iterator generators, that is, an iterator generator 
which recursively produces iterator generators. 

Learn that your implementation of an iterator object (an iterator class) can 
"refresh" itself and learn at least one way to do this. 


Definitions: 


Iterator - And iterator is an object that satisfies (implements) the iterator protocol. 
Iterator protocol - An object implements the iterator protocol if it implements both 
anext () andan__iter___() method which satisfy these rules: (1) the 
___iter__() method must return the iterator; (2) the next () method should 
return the next item to be iterated over and when finished (there are no more 
items) should raise the StopTteration exception. The iterator protocol is 
described at Iterator Types -- 
http://docs.python.org/library/stdtypes.html#iterator-types. 

Iterator class - A class that implements (satisfies) the iterator protocol. In 
particular, the class implements next () and__iter___() methods as 
described above and in Iterator Types -- 
http://docs.python.org/library/stdtypes.html#iterator-types. 

(Iterator) generator function - A function (or method) which, when called, returns 
an iterator object, that is, an object that satisfies the iterator protocol. A function 
containing a yield statement automatically becomes a generator. 

Generator expression - An expression which produces an iterator object. 
Generator expressions have a form similar to a list comprehension, but are 
enclosed in parentheses rather than square brackets. See example below. 


A few additional basic points: 


A function that contains a yield statement is a generator function. When called, it 
returns an iterator, that is, an object that provides next () and__iter___() 
methods. 

The iterator protocol is described here: Python Standard Library: Iterator Types -- 
http://docs.python.org/library/stdtypes.html#iterator-types. 
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e Aclass that defines both a next () method anda__iter___() method satisfies 
the iterator protocol. So, instances of such a class will be iterators. 

e Python provides a variety of ways to produce (implement) iterators. This section 
describes a few of those ways. You should also look at the iter () built-in 
function, which is described in The Python Standard Library: Built-in Functions: 
iter() -- http://docs.python.org/library/functions.html#iter. 

e An iterator can be used in an iterator context, for example in a for statement, in a 
list comprehension, and in a generator expression. When an iterator is used in an 
iterator context, the iterator produces its values. 

This section attempts to provide examples that illustrate the generator/iterator pattern. 


Why is this important? 


e Once mastered, it is a simple, convenient, and powerful programming pattern. 

e [thas many and pervasive uses. 

e Ithelps to lexically separate the producer code from the consumer code. Doing so 
makes it easier to locate problems and to modify or fix code in a way that is 
localized and does not have unwanted side-effects. 

e Implementing your own iterators (and generators) enables you to define your own 
abstract sequences, that is, sequences whose composition are defined by your 
computations rather than by their presence in a container. In fact, your iterator can 
calculate or retrieve values as each one is requested. 

Examples - The remainder of this section provides a set of examples which implement 
and use iterators. 


2.3.1 Example - A generator function 


This function contains a yield statement. Therefore, when we call it, it produces an 
iterator: 


def generatelItems (seq): 
for item in seq: 
yield ‘item: %s' % item 


anIter = generatelItems([]) 
Deine ete (ankeer) "7 dima(aniter) 
anIter = generateItems ([111, 222, 333]) 
for x in anlIter: 
joneskings, =< 
anIter = generatelItems(['aaa', 'bbb', 'ccc']) 
print anIter.next () 
print anIter.next () 
print anIter.next () 
print anIter.next () 


Running this example produces the following output: 
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Glilia (venallbiceie)) 2° (py Glktais’s ee Cledvatts eis vp PL adoe prs 
' getattribute__', 
bo loeisian ALLA tee UE ieee YE eee Wee reduce__', 
Va HeCuce lex 27, Ve sepriu';, Versetari rie Vasrrla!) “oieramet, 
Jisfak_iaiobavaseiae: Vp Vier ece Y | 
ieemc hal 
item: 222 
item: 333 
item: aaa 
item: bbb 
LeeMc Ce 
Traceback (most recent call last): 

File "iterator_generator.py", line 14, in ? 

print anIter.next () 

StopIteration 


Notes and explanation: 


e The value returned by the call to the generator (function) is an iterator. It obeys 
the iterator protocol. That is, dir (anIter) shows that it has both 
__ iter __() and next () methods. 

e Because this object is an iterator, we can use a for statement to iterate over the 
values returned by the generator. 

e Wecan also get its values by repeatedly calling the next () method, until it 
raises the StopIteration exception. This ability to call the next method enables us 
to pass the iterator object around and get values at different locations in our code. 

e Once we have obtained all the values from an iterator, it is, in effect, "empty" or 
"exhausted". The iterator protocol, in fact, specifies that once an iterator raises the 
StopIteration exception, it should continue to do so. Another way to say this is 
that there is no "rewind" operation. But, you can call the the generator function 
again to get a "fresh" iterator. 

An alternative and perhaps simpler way to create an interator is to use a generator 
expression. This can be useful when you already have a collection or iterator to work 
with. 


Then following example implements a function that returns a generator object. The effect 
is to generate the objects in a collection which excluding items in a separte collection: 


DATA = [ 

'lemon', 
aime; 
"grape', 
‘apple', 
"pear', 
"watermelon', 
"canteloupe', 
"honeydew', 
"orange', 
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"Graperruxt *, 


] 


def make_producer (collection, excludes): 
gen = (item for item in collection if item not in excludes) 
Bee Misi Geil 


Ce ces) & 
iterl = make_producer (DATA, ('apple', 'orange', ‘honeydew', )) 
joneabime, Vass! es” shietSae tl 


heen Ghiayislye alin) alyeretia ll 
joueatighe, sgsaiblokie 


test () 


When run, this example produces the following: 


S python workbook063.py 

<generator object <genexpr> at 0x7fb3d0flbc80> 
lemon 

lime 

grape 

pear 

watermelon 

canteloupe 

grapefruit 


Notes: 


e A generator expression looks almost like a list comprehension, but is surrounded 
by parentheses rather than square brackets. For more on list comprehensions see 
section Example - A list comprehension. 

e The make_producer function returns the object produced by the generator 
expression. 


2.3.2 Example - A class containing a generator method 


Each time this method is called, it produces a (new) iterator object. This method is 
analogous to the iterkeys and itervalues methods in the dictionary built-in object: 


# 
# A class that provides an iterator generator method. 
# 


class Node: 


def __ init__(self, name='<noname>', value='<novalue>', 
children=None): 
self.name = name 
self.value = value 
self.children = children 


if children is None: 
sellig ,claakicleein = |] 
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# 


# 


# 


else: 


sel 
set_name (self 
get_name (sel 


SCiemvicl 
GeeLva 


wee 


i e@lgislsl 


a 


Hayes 


dren 
name): 


@laiat Ielie nal 
self.name 
return self.name 


f, 
Eyes 


ue (sel 
ue (sel 


@l alan se] 


value): self.va 
return self.val 


Tbemeha einen (sielitz \: 
Toi Claat) 


if 5 Cleat Ielkeeiaye 


edlay ena 


# Print information on this node and walk over all children and 


grandchildren 
def walk(self, level=0) 
pring 'esname: os value: cs" 3 ( 
get_filler(level), self.get_name(), self.get_value(), ) 
foe halo an seit arenchalldren ()i¢ 
child.walk(level + 1) 


# An function that is the equivalent of the walk() 


method in 


# class Node. 
# 
def walk(node, level=0): 
print '%Ssname: %S value: %s' % ( 
get_filler(level), node.get_name(), node.get_value(), ) 
Om haskell ane mMeder ate relmaalkeneeta()\r 
walk(child, level + 1) 
def get_filler(level): 
return ' '" * level 
def test(): 
aly = Node(onilipenc te! aii) 
a6 = Node('fred' "666') 
a5 = Node('tellie', '555') 
a4 = Node('daniel', '444') 
as) — Neode(YeGanri '3383") [a4 as) 
a2 = Node('bill', '222', [a6, a7]) 
al > Nede(Pailicet = “ial yaa es asi) 
# Use the walk method to walk the entire tree. 
print 'Using the method:' 
al.walk () 
jonenlinyen Uae ci0l 
# Use the walk function to walk the entire tree. 
Deine Using, pher tuncelon= 
walk (al) 
test () 
nning this example produces the following output: 


name: 


alice 


Using the method: 
value: 


alealale 
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name: bill value: 222 
name: fred value: 666 
name: gilbert value: 777 
name: carl value: 333 
name: daniel value: 444 
names “elie “values 555 
Using the function: 
name: alice value: 111 
name: bill Values 222 
name: fred value: 666 
name: gilbert value: 777 
name: carl value: 333 
name: daniel value: 444 
name: ellie value: 555 


Notes and explanation: 


This class contains a method iterchildren which, when called, returns an iterator. 
The yield statement in the method iterchildren makes it into a generator. 
The yield statement returns one item each time it is reached. The next time the 


iterator object is "called" it resumes immediately after the yield statement. 


A function may have any number of yield statements. 
A for statement will iterate over all the items produced by an iterator object. 
This example shows two ways to use the generator, specifically: (1) the walk 


method in the class Node and (2) the walk function. Both call the generator 
iterchildren and both do pretty much the same thing. 


2.3.3 Example - An iterator class 


This class implements the iterator protocol. Therefore, instances of this class are iterators. 
The presence of the next () and__iter___() methods means that this class 
implements the iterator protocol and makes instances of this class iterators. 


Note that when an iterator is "exhausted" it, normally, cannot be reused to iterate over the 
sequence. However, in this example, we provide a refresh method which enables us to 
"rewind" and reuse the iterator instance: 


# 


# An iterator class that does *not* use 


> yarewor 


if self.idx >= 


tf This iterator produces every other item in a sequence. 
# 
class IteratorExample: 
def __init__(self, seq): 
self.seq = seg 
self.idx = 0 
def next (self): 
selt.idx += il 


len(self.seq): 
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if 


raise StopIteration 
value = self.seq[self.idx] 
self.idx += 1 
return value 
det later (sehr) = 
return self 
def refresh(self): 


self.idx = 0 


test_iteratorexample(): 
a = IteratorExample('edcba') 
Ia es akigvest a 
fone aye 
(iaulinc | ' 
a.refresh() 
Om xe seme cus 
joneakione, 
pring, t=! 930 
a = IteratorExample('abcde') 
raya 
jouealiane, Gl () 
jONeaLING “al () 
print a.next () 
a () 
a OQ) 


joneabione 
[eae 
print a.next () 
except StopIteration, e: 
pring Stopping’, e 


test_iteratorexample() 


nning this example produces the following output: 


d 


stopping 


Notes and explanation: 


e The next method must keep track of where it is and what item it should produce 


next. 

Alert: The iterator protocol has changed slightly in Python 3.0. In particular, the 
next () method has been renamed to__next___() . See: Python Standard 
Library: Iterator Types -- 
http://docs.python.org/3.0/library/stdtypes.html#iterator-types. 


Page 103 


A Python Book 


2.3.4 Example - An iterator class that uses yield 


There may be times when the next method is easier and more straight-forward to 
implement using yield. If so, then this class might serve as an model. If you do not feel 
the need to do this, then you should ignore this example: 


# 

# An iterator class that uses “‘yield’>. 

tf This iterator produces every other item in a sequence. 
# 


class YieldIteratorExample: 
def __init__(self, seq): 
self.seq = seq 
self.iterator = self._next () 
self.next = self.iterator.next 
def _next (self): 
flag = 0 
for x in self.seq: 
aie 1B LENGE 
flag = 0 
yield x 
else: 
lieve) = 
def soiter (self): 
return self.iterator 
def refresh(self): 
self.iterator = self._next () 
self.next = self.iterator.next 


1 


def test_yielditeratorexample(): 
a = YieldIteratorExample('edcba') 
OIG 7X aay fale 


a.refresh() 
Om x Sane ce 
[OieLING 2 
forednie, te) 
a = YieldIteratorExample('abcde') 
qeqanyn 


jie aL ane 
jOneaLine. 
print 
joneabighe 
joneatione 
print a.next () 
except StopIteration, e: 
Dring VSstopping', 


o ooo 
3) 
0) 
* 
a 


test_yielditeratorexample () 


Running this example produces the following output: 
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stopping 


Notes and explanation: 


e Because the _next method uses yield, calling it (actually, calling the iterator 
object it produces) in an iterator context causes it to be "resumed" immediately 
after the yield statement. This reduces bookkeeping a bit. 

e However, with this style, we must explicitly produce an iterator. We do this by 
calling the _next method, which contains a yield statement, and is therefore a 
generator. The following code in our constructor (__init___) completes the 
set-up of our class as an iterator class: 


self.iterator = self._next() 
self.next = self.iterator.next 


Remember that we need both__iter___() and next () methods in 
YieldIteratorExamp1e to satisfy the iterator protocol. The __iter___() 
method is already there and the above code in the constructor creates the next () 
method. 


2.3.5 Example - A list comprehension 


A list comprehension looks a bit like an iterator, but it produces a list. See: The Python 
Language Reference: List displays -- 
http://docs.python.org/reference/expressions.html#list-displays for more on list 
comprehensions. 


Here is an example: 


En [4s det fie) = 
Pechreks is(etmlsial SS) 


eta 5s Iseeic i = alal 2 sss) 

iia GJ bse se (Gx) eeie So sviay aes) | 
Ieiay | FA Se yoneakiane, llskisic 2 

LSa,, 86, SS] 


2.3.6 Example - A generator expression 


A generator expression looks quite similar to a list comprehension, but is enclosed in 
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parentheses rather than square brackets. Unlike a list comprehension, a generator 
expression does not produce a list; it produces an generator object. A generator object is 
an iterator. 


For more on generator expressions, see The Python Language Reference: Generator 
expressions -- http://docs.python.org/reference/expressions.html#generator-expressions. 


The following example uses a generator expression to produce an iterator: 


mylist = range(10) 


lane se (2) 6 
iseNewlsin Sess} 


genexpr = (f(x) for x in mylist) 


for x in genexpr: 
joie slins 


Notes and explanation: 


e The generator expression (f(x) for x in mylist) produces an iterator object. 
e Notice that we can use the iterator object later in our code, can save it in a data 
structure, and can pass it to a function. 


2.4 Unit Tests 


Unit test and the Python unit test framework provide a convenient way to define and run 
tests that ensure that a Python application produces specified results. 


This section, while it will not attempt to explain everything about the unit test framework, 
will provide examples of several straight-forward ways to construct and run tests. 


Some assumptions: 


e Weare going to develop a software project incrementally. We will not implement 
and release all at once. Therefore, each time we add to our existing code base, we 
need a way to verify that our additions (and fixes) have not caused new problems 
in old code. 

e Adding new code to existing code will cause problems. We need to be able to 
check/test for those problems at each step. 

e As we add code, we need to be able to add tests for that new code, too. 


2.4.1 Defining unit tests 


2.4.1.1 Create a test class. 


In the test class, implement a number of methods to perform your tests. Name your test 
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methods with the prefix "test". Here is an example: 


import unittest 


class MyTest (unittest.TestCase): 
def test_one(self): 
# some test code 
pass 
def test_two(self): 
# some test code 
pass 


Create a test harness. Here is an example: 


import unittest 


# make the test suite. 

def suite(): 
loader = unittest.TestLoader () 
testsuite = loader.loadTestsFromTestCase (MyTest) 
return testsuite 


# Make the test suite; run the tests. 


Glee icSsic ()) & 
testsuite = suite() 
runner = unittest.TextTestRunner(sys.stdout, verbosity=2) 
result = runner.run(testsuite) 


Here is a more complete example: 


IMOowe Enis) Sei@akoveIlO,  Sieseuhare) 
import unittest 
import webserv_example_heavy_sub 


# A comparison function for case-insenstive sorting. 
def mycmpfunc(argl, arg2): 
return cmp(string.lower(argl), string.lower(arg2) ) 


class XmlTest (unittest.TestCase): 
def test_import_exportl (self): 


alta ee —— elke (Mme Sitede salem. sore ees Use) 
inContent = inFile.read() 
inFile.close() 


doc = webserv_example_heavy_sub.parseString(inContent) 
CuUEH Ile = — Stra nglOr St rrncOr) 

outFile.write('<?xml version="1.0" ?>\n") 

doc.export (outFile, 0) 

outContent = outFile.getvalue() 

outFile.close() 

self.failUnless (inContent == outContent) 


# make the test suite. 
def suite(): 
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loader = unittest.TestLoader () 

# Change the test method prefix: test --> trial. 
loader.testMethodPrefix = 'trial' 

Change the comparison function that determines the order of 


He tk 


tests. 
loader.sortTestMethodsUsing = mycmpfunc 
testsuite = loader.loadTestsFromTestCase (XmlTest) 
return testsuite 


te 


# Make the test suite; run the tests. 
def test_main(): 


testsuite = suite() 
runner = unittest.TextTestRunner(sys.stdout, verbosity=2) 
result = runner.run(testsuite) 

SLE name == "_ main __": 


test_main() 


nning the above script produces the following output: 


test_import_export (__main__.XmlTest) ... ok 


Ran 1 test in 0.035s 


OK 


A few notes on this example: 


e This example tests the ability to parse an xml document testl_in.xml and export 
that document back to XML. The test succeeds if the input XML document and 
the exported XML document are the same. 

e The code which is being tested parses an XML document returned by a request to 
Amazon Web services. You can learn more about Amazon Web services at: 
http://www.amazon.com/webservices. This code was generated from an XML 
Schema document by generateDS.py. So we are in effect, testing generateDS.py. 
You can find generateDS.py at: 
http://http://www.davekuhIman.org/#generateds-py. 

e Testing for success/failure and reporting failures -- Use the methods listed at 
http://www.python.org/doc/current/lib/testcase-objects.html to test for and report 
success and failure. In our example, we used "self.failUnless(inContent == 
outContent)" to ensure that the content we parsed and the content that we 
exported were the same. 

e Add additional tests by adding methods whose names have the prefix "test". If 
you prefer a different prefix for tests names, add something like the following to 
the above script: 


loader.testMethodPrefix = 'trial' 
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e By default, the tests are run in the order of their names sorted by the cmp 


function. So, if needed, you can control the order of execution of tests by 
selecting their names, for example, using names like test_1_checkderef, 
test_2_checkcalc, etc. Or, you can change the comparison function by adding 
something like the following to the above script: 


loader.sortTestMethodsUsing = mycmpfunc 


As a bit of motivation for creating and using unit tests, while developing this example, I 
discovered several errors (or maybe "special features") in generateDS.py. 


2.5 Extending and embedding Python 


2.5.1 


Introduction and concepts 


Extending vs. embedding -- They are different but related: 


e Extending Python means to implement an extension module or an extension type. 


An extension module creates a new Python module which is implemented in 
C/C++. From Python code, an extension module appears to be just like a module 
implemented in Python code. An extension type creates a new Python (built-in) 
type which is implemented in C/C++. From Python code, an extension type 
appears to be just like a built-in type. 

Embedding Python, by contrast, is to put the Python interpreter within an 
application (i.e. link it in) so that the application can run Python scripts. The 
scripts can be executed or triggered in a variety of ways, e.g. they can be bound to 
keys on the keyboard or to menu items, they can be triggered by external events, 
etc. Usually, in order to make the embedded Python interpreter useful, Python is 
also extended with functions from the embedding application, so that the scripts 
can call functions that are implemented by the embedding C/C++ application. 


Documentation -- The two important sources for information about extending and 
embedding are the following: 


Extending and Embedding the Python Interpreter -- 
http://www.python.org/doc/current/ext/ext.html 
Python/C API Reference Manual -- 
http://www.python.org/doc/current/api/api.html 


Types of extensions: 


Extension modules -- From the Python side, it appears to be a Python module. 
Usually it exports functions. 

Extension types -- Used to implement a new Python data type. 

Extension classes -- From the Python side, it appears to be a class. 
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Tools -- There are several tools that support the development of Python extensions: 


e SWIG -- Learn about SWIG at: http://www.swig.org 


e Pyrex -- Learn about Pyrex at: 


http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/ 
e There is also Cython, which seems to be an advanced version of, or at least an 
alternative to Pyrex. See: Cython - C Extensions for Python -- 


http://www.cython.org/ 


2.5.2 Extension modules 


Writing an extension module by hand -- What to do: 


e Create the "init" function -- The name of this function must be "init" followed by 


the name of the module. Every extension module must have such a function. 
e Create the function table -- This table maps function names (referenced from 
Python code) to function pointers (implemented in C/C++). 


e Implement each wrapper function. 
Implementing a wrapper function -- What to do: 


1. Capture the arguments with PyArg_ParseTuple. The format string specifies how 
arguments are to be converted and captured. See 1.7 Extracting Parameters in 
Extension Functions. Here are some of the most commonly used types: 

o Use "i", "s", "f", etc to convert and capture simple types such as integers, 


strings, floats, etc. 


o Use "O" to get a pointer to Python "complex" types such as lists, tuples, 


dictionaries, etc. 


o Use items in parentheses to capture and unpack sequences (e.g. lists and 


tuples) of fixed length. Example: 


ti (i PyArcquPacseluple (args, (abn) (in) Wy esc (Sy, 
é&width, &height) ) 
{ 
return NULL; 
je aioe alee 8 
A sample call might be: 
lowerLeft = (xl, yl) 
extent = (widthl, height1) 
scan(lowerLeft, extent) 


o Use ":aName" (colon) at the end of the format string to provide a function 


name for error messages. Example: 


if (!PyArg_ParseTuple(args, 
é&pythoninstance) ) 
{ 


"O:setContentHandler", 
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return NULL; 
Tey ee tec 


o Use ";an error message" (semicolon) at the end of the format string to provide 
a string that replaces the default error message. 

o Docs are available at: http://www.python.org/doc/current/ext/parseTuple.html. 

Write the logic. 

Handle errors and exceptions -- You will need to understand how to (1) clearing 

errors and exceptions and (2) Raise errors (exceptions). 

o Many functions in the Python C API raise exceptions. You will need to check 
for and clear these exceptions. Here is an example: 


char * message; 
int messageNo; 


message — aN Ui, 

messageNo = -1; 

/* Is the argument a string? 

x/ 

if (! PyArg_ParseTuple(args, "s", &message) ) 

{ 
fee Mie Vs ine &) Sivieiow, Clee tele erence 
* Then try to get a message number (an 


integer). 
ais 
PyErr_Clear(); 
if (! PyArg_ParseTuple(args, "i", &messageNo) ) 


{ 
fo) 
° 
° 


o Youcan also raise exceptions in your C code that can be caught (in a 
"try:except:" block) back in the calling Python code. Here is an example: 


if (n == 0) 
{ 


PyErr_SetString (PyExc_ValueError, "Value must 
not be zero"); 

return NULL; 
} 


See Include/pyerrors.h in the Python source distribution for more 
exception/error types. 

o And, you can test whether a function in the Python C API that you have called 
has raised an exception. For example: 


ih (Pyare Occummed())) 

{ 
/* An exception was raised. 
* Do something about it. 
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For more documentation on errors and exceptions, see: 
http://www.python.org/doc/current/api/exceptionHandling.html. 


4. Create and return a value: 


o For each built-in Python type there is a set of API functions to create and 


manipulate it. See the "Python/C API Reference Manual" for a description of 
these functions. For example, see: 

= = http://www.python.org/doc/current/api/intObjects. html 
http://www.python.org/doc/current/api/stringObjects.html 
http://www.python.org/doc/current/api/tupleObjects.html 
http://www.python.org/doc/current/api/listObjects.html 

http://www. python.org/doc/current/api/dictObjects.html 

Ete. 

The reference count -- You will need to follow Python's rules for reference 
counting that Python uses to garbage collect objects. You can learn about 
these rules at http://www.python.org/doc/current/ext/refcounts.html. You will 
not want Python to garbage collect objects that you create too early or too late. 
With respect to Python objects created with the above functions, these new 
objects are owned and may be passed back to Python code. However, there 
are situations where your C/C++ code will not automatically own a reference, 
for example when you extract an object from a container (a list, tuple, 
dictionary, etc). In these cases you should increment the reference count with 
Py_INCREF. 


2.5.3 SWIG 


Note: Our discussion and examples are for SWIG version 1.3 


SWIG will often enable you to generate wrappers for functions in an existing C function 
library. SWIG does not understand everything in C header files. But it does a fairly 
impressive job. You should try it first before resorting to the hard work of writing 
wrappers by hand. 


More information on SWIG is at http://www.swig.org. 


Here are some steps that you can follow: 


1. 


Create an interface file -- Even when you are wrapping functions defined in an 
existing header file, creating an interface file is a good idea. Include your existing 
header file into it, then add whatever else you need. Here is an extremely simple 
example of a SWIG interface file: 
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%{ 


% } 


#incl 


es dL iaueldL 


ude 


ude 


Smodule MyLibrary 


"My Eabomgetioys ian 


HRV albyall oparstaeny/ & la\ 


Comments: 


o The "%{" and "%}" brackets are directives to SWIG. They say: "Add the code 
between these brackets to the generated wrapper file without processing it. 

o The "%include" statement says: "Copy the file into the interface file here. In 
effect, you are asking SWIG to generate wrappers for all the functions in this 
header file. If you want wrappers for only some of the functions in a header 
file, then copy or reproduce function declarations for the desired functions 
here. An example: 


%{ 
#include "MyLibrary.h" 


%} 


Smodule MyLibrary 


int calcArea(int width, int height); 


imi Cale Volumes (smite ieaidatuais)) > 


This example will generate wrappers for only two functions. 
o Youcan find more information about the directives that are used in SWIG 
interface files in the SWIG User Manual, in particular at: 
m = http://www.swig.org/Doc1.3/Preprocessor.html 
m = http://www.swig.org/Doc1.3/Python.html 


2. Generate the wrappers: 


swig -python MyLibrary.i 


3. Compile and link the library. On Linux, you can use something like the following: 


SiGe =e) INhVALaLloresiey 

Gee {ee l/s locally inelude/pytnoen2. 5) Mylo tcenay mnt ionic 
gcc -shared MyLibrary.o MyLibrary_wrap.o -o 
_MyLibrary.so 


Note that we produce a shared library whose name is the module name prefixed 
with an underscore. SWIG also generates a .py file, without the leading 
underscore, which we will import from our Python code and which, in turn, 
imports the shared library. 

4. Use the extension module in your python code: 


Python 2 Soi) (Gli Apr Zor Zi00s,5 2036209) 
[GCC 2.95.4 20011002 (Debian prerelease)] on linux2 
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ivpe “helio “copycrghely, “credits ior 
more information. 

>>> import MyLibrary 

2>> Myiiiibrany icalcArea (4.07 5.10!) 

2050 


Wikaleeiase ieee 


Here is a makefile that will execute swig to generate wrappers, then compile and link the 


extension. 
CFLAGS = -I/usr/local/include/python2.3 
all: _MyLibrary.so 
_MyLibrary.so: MyLibrary.o MyLibrary_wrap.o 
gcc -shared MyLibrary.o MyLibrary_wrap.o -o _MyLibrary.so 
MyLibrary.o: MyLibrary.c 
gcc -c MyLibrary.c -o MyLibrary.o 
MyLibrary_wrap.o: MyLibrary_wrap.c 
gcc -c ${ CFLAGS} MyLibrary_wrap.c -o MyLibrary_wrap.o 
MyLibrary_wrap.c: MyLibrary.i 
swig -python MyLibrary.i 
clean: 
rm -f MyLibrary.py MyLibrary.o MyLibrary_wrap.c 
MyLibrary_wrap.o _MyLibrary.so 


Here is an example of running this makefile: 


S$ make -f MyLibrary_makefile clean 

rm -f MyLibrary.py MyLibrary.o MyLibrary_wrap.c \ 
MyLibrary_wrap.o _MyLibrary.so 

S$ make -f MyLibrary_makefile 

CEG —© Whlillseeiay/,@ 6) Myjikabloeaiiey 50) 

swig -python MyLibrary.i 


MyLibrary_wrap.o 


gece) —e —l/usr/lecaly/inelude/python2=3) Myiltoranryawircapae =O 


gcc -shared MyLibrary.o MyLibrary_wrap.o -o _MyLibrary.so 


And, here are C source files that can be used in our example. 


MyLibrary.h: 


/* MyLibrary.h 
“ay 
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float calcArea(float width, float height); 
float calcVolume (float radius); 


int getVersion(); 


int getMode(); 


MyLibrary.c: 


j* Mylibrary.¢ 
ay 


Float calcArea(float width, float height) 


return (width * height); 


Elloak iGaleVolume(fileatk cadaiuis)) 


weeinbheim (si, 14 = sesicliiois; ~ iecichhiuis})\p 


int getVersion () 


ee we leas 


int getMode () 


return 1; 


2.5.4 Pyrex 


Pyrex is a useful tool for writing Python extensions. Because the Pyrex language is 
similar to Python, writing extensions in Pyrex is easier than doing so in C. Cython 
appears to be the a newer version of Pyrex. 


More information is on Pyrex and Cython is at: 


e Pyrex -- http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/ 
e Cython - C Extensions for Python -- http://www.cython.org/ 
Here is a simple function definition in Pyrex: 


# python_201_pyrex_string.pyx 
LMP Obs ssie aemG) 
def formatString(object sl, object s2): 


si — String.strip (si) 
SZ, = Biceane/ Sic lon(s 2) 
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33 Ucn || || sees) 2 (sil, s2)) 
s4=s3 * 4 
return s4 


And, here is a make file: 


CFLAGS = -DNDEBUG -0O3 -Wall -Wstrict-prototypes -fPIC \ 
-T/usr/local/include/python2.3 


all: python 201 ipyrex string. so 


python_201_pyrex_string.so: python_201_pyrex_string.o 
gcc -shared python_201_pyrex_string.o -o 
python_201_pyrex_string.so 


python_201_pyrex_string.o: python_201_pyrex_string.c 
Geee Se 1S CulLAGS) woycenonm 0 pve xms can Gme! © 
python_201_pyrex_string.o 


python_201_pyrex_string.c: python_201_pyrex_string.pyx 
pyrexc python_201_pyrex_string.pyx 


eileen: 
tif pyenon 20) ipyrexustiang. so PpyEnon 20 le pynexuskringao \ 
python_201_pyrex_string.c 


Here is another example. In this one, one function in the .pyx file calls another. Here is 
the implementation file: 


# python_201_pyrex_primes.pyx 


def showPrimes (int kmax): 
plist = primes (kmax) 
Ione joy akigy joullsiisie & 
DLiMe Vorimes: scl so 


cdef primes(int kmax): 
edert ant am, ky, al 
cdef int p[1000] 
result = [] 
if kmass > WOOO 
kmax = 1000 


k = 0 
n= 2 
while k < kmax: 
i = 0 
(welalies aL <1 ehalcl sey = O(a) <= Oe 
see ee 
if i == k: 
Ose) = at 
k=k ed 
result.append (n) 
ia = toy ap dL 
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return result 


And, here is a make file: 


#CFLAGS = -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC # 
-I/usr/local/include/python2.3 CFLAGS = -DNDEBUG 
-I/usr/local/include/python2.3 


all: python_201_pyrex_primes.so 

python_201_pyrex_primes.so: python_201_pyrex_primes.o 

gcc -shared python_201_pyrex_primes.o -o python_201_pyrex_primes.so 
python_201_pyrex_primes.o: python_201_pyrex_primes.c 

gcc -c ${ CFLAGS} python_201_pyrex_primes.c -o python_201_pyrex_primes.o 
python_201_pyrex_primes.c: python_201_pyrex_primes.pyx 

pyrexc python_201_pyrex_primes.pyx 
clean: 

rm -f python_201_pyrex_primes.so python_201_pyrex_primes.o 

python_201_pyrex_primes.c 


Here is the output from running the makefile: 


S$ make -f python_201_pyrex_makeprimes clean 

rm -f£f python_201_pyrex_primes.so python_201_pyrex_primes.o \ 
python_201_pyrex_primes.c 

S$ make -f python_201_pyrex_makeprimes 

pyrexc python_201_pyrex_primes.pyx 

gcee oe  DNDEBUGe liste Moca imellicley ex tenomAzns 

python_201_pyrex_primes.c -o python_201_pyrex_primes.o 

gcc -shared python_201_pyrex_primes.o -o python_201_pyrex_primes.so 


Here is an interactive example of its use: 


S python 

Python 2a3oh (Hl, Ape 25 20037 20% 36-109) 

[GCC 2.95.4 20011002 (Debian prerelease) ] on linux2 

Type "help", "copyright", "credits" or "license" for more 
information. 

>>> import python_201_pyrex_primes 

>>> dir(python_201_pyrex_primes) 

poppe sino Yo eloyel Sy esky inchs Ve Vclorenyesenaivers | 
>>> python_201_pyrex_primes.showPrimes (5) 

prime: 2 

jOueamiess 5) 

jOue amie 5) 

prime: 7 
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prime: 11 


This next example shows how to use Pyrex to implement a new extension type, that is a 
new Python built-in type. Notice that the class is declared with the cdef keyword, which 


tells Pyrex to generate the C implementation of a type instead of a class. 


Here is the implementation file: 


# python_201_pyrex_clsprimes.pyx 


"""An implementation of primes handling class 
for a demonstration of Pyrex. 


ww 


cdef class Primes: 
HN CULSISS Crome Slaligubigte; “sciblakewalfoualis;  icei2 
handling primes. 


ww 


def showPrimes(self, int kmax): 
"Show a range of primes. 
Use the method primes() to generate the primes. 
ww 
plist = self.primes (kmax) 
iBOne jos SLi’ joulaisicrs 
Prine “primes sad 4p 


def primes(self, int kmax): 
"""Generate the primes in the range 0 —- kmax. 
wo 
det Ginty a, ks) a 
cdef int p[1000] 
result = [] 
if kmas > O00: 
kmax = 1000 


k = 0 
ia = 
while k < kmax: 
i = 0 
while i< k and n % p[i] <> 0: 
i=i+tl 
if i == k: 
Oils) = w 
k=k+41 


result.append (n) 
io = ioe ap AL 
return result 


And, here is a make file: 


CFLAGS = -DNDEBUG -I/usr/local/include/python2.3 


all: python_201_pyrex_clsprimes.so 
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python_201_pyrex_clsprimes.so: python_201_pyrex_clsprimes.o 
gcc -shared python_201_pyrex_clsprimes.o -o 
python_201_pyrex_clsprimes.so 


python_201_pyrex_clsprimes.o: python_201_pyrex_clsprimes.c 
Gee eo (CHLAGS) Voy enone lpycexuclsprimesncl—© 
python_201_pyrex_clsprimes.o 


python_201_pyrex_clsprimes.c: python_201_pyrex_clsprimes.pyx 
pyrexc python_201_pyrex_clsprimes.pyx 


euicann: 
rm -£ python_201_pyrex_clsprimes.so 
python_201_pyrex_clsprimes.o \ 
python_201_pyrex_clsprimes.c 


Here is output from running the makefile: 


S$ make -f python_201_pyrex_makeclsprimes clean 

rm -f python_201_pyrex_clsprimes.so python_201_pyrex_clsprimes.o \ 
python_201_pyrex_clsprimes.c 

S$ make -f python_201_pyrex_makeclsprimes 

pyrexc python_201_pyrex_clsprimes.pyx 

eS —C SDINIDINBWE 1 / sie keesiiL/ tne iluicley/joxae iavenaZ) ,, 3} 

python_201_pyrex_clsprimes.c -o python_201_pyrex_clsprimes.o 

gcc -shared python_201_pyrex_clsprimes.o -o 

python_201_pyrex_clsprimes.so 


And here is an interactive example of its use: 


S) JeNAelaenal 

IWiclgeinl 25 Sioik (Gail Nels 23) ZOOS 210 esis 50) 

[GCC 2.95.4 20011002 (Debian prerelease) ] on linux2 

Type "help", "copyright", "credits" or "license" for more 
information. 

>>> import python_201_pyrex_clsprimes 

>>> dir (python_201_pyrex_clsprimes) 

[Merimest) Sub Use isie sy clo cian rem) Silene al 
>>> primes = python_201_pyrex_clsprimes.Primes () 

>>> dir (primes) 


Ries eukasis 7 eC eilaretars i Chere! V7, ‘os acecrarrributes oF 
Veehashwe, 

i ALAC Me CS me Woes KBeduces 7) “es ceduiceres repr ne 
'  setattr__', '__str__', 'primes', 'showPrimes'] 


>>> primes.showPrimes (4) 
prime: 2 
ues eS) 
joveauinnes3 5) 
prime: 7 


Documentation -- Also notice that Pyrex preserves the documentation for the module, the 
class, and the methods in the class. You can show this documentation with pydoc, as 
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follows: 


S pydoc python_201_pyrex_clsprimes 


Or, in Python interactive mode, use: 


S python 

Python 2 solely Apr 2a) A003. 20K 3:6'-10)9)) 

[GCC 2.95.4 20011002 (Debian prerelease) ] on linux2 

Type "help", "copyright", "credits" or "license" for more 
information. 

>>> import python_201_pyrex_clsprimes 

>>> help (python_201_pyrex_clsprimes) 


2.5.5 SWIG vs. Pyrex 
Choose SWIG when: 


e You already have an existing C or C++ implementation of the code you want to 
call from Python. In this case you want SWIG to generate the wrappers. But note 
that Cython promises to enable you to quickly wrap and call functions 
implemented in C. 

e You want to write the implementation in C or C++ by hand. Perhaps, because you 
think you can do so quickly, for example, or because you believe that you can 
make it highly optimized. Then, you want to be able to generate the Python 
(extension) wrappers for it quickly. 

Choose Pyrex when: 


e You do not have a C/C++ implementation and you want an easier way to write 
that C implementation. Writing Pyrex code, which is a lot like Python, is easier 
than writing C or C++ code by hand). 

e You start to write the implementation in C, then find that it requires lots of calls to 
the Python C API, and you want to avoid having to learn how to do that. 


2.5.6 Cython 
Here is a simple example that uses Cython to wrap a function implemented in C. 


First the C header file: 


/ Seis waco eee), 


int calculate(int width, int height); 


And, the C implementation file: 


[Sees eile 227/ 
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#include "test_c_lib.h" 


int calculate(int width, int height) 
{ 


Sbigue, | iaticuulllie r 
result = width * height * 3; 
return result; 


} 


Here is a Cython file that calls our C function: 


# test_c.pyx 


# Declare the external C function. 
Geli xcuesein incom Wiese Ilorin 
int calculate(int width, int height) 


def test(w, h): 
# Call the external C function. 
result = calculate(w, h) 
print ‘result from calculate: %d' % result 


We can compile our code using this script (on Linux): 


#!/bin/bash -x 

GWAElaNoia TeSisie Tels joe 

Cee Se SPC S/S) ikeesil / salerlwrele/jeycincmA,' =e) Sse C56 wesc © .C 
gece -c -fPIC -I/usr/local/include/python2.6 -o test_c_lib.o 

He stbuel Lube 
gee), -shareds—rPie —h/usr/locall/ancilude/pytnonzg..6 )—o testac. so 
EeGSELG 70 test rel lib 


Here is a small Python file that uses the wrapper that we wrote in Cython: 


# run_test_c.py 

import test_c 

def test(): 
test_c.test (4, 5) 
Gest o.tese (12, 1:5) 


Stag name == ' | main Use 
test () 


And, when we run it, we see the following: 


S) jehvielaehal seblial Teese (eon 
result from calculate: 60 
result from calculate: 540 
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2.5.7 Extension types 
The goal -- A new built-in data type for Python. 


Existing examples -- Objects/listobject.c, Objects/stringobject.c, Objects/dictobject.c, etc 
in the Python source code distribution. 


In older versions of the Python source code distribution, a template for the C code was 
provided in Objects/xxobject.c. Objects/xxobject.c is no longer included in the Python 
source code distribution. However: 


e The discussion and examples for creating extension types have been expanded. 
See: Extending and Embedding the Python Interpreter, 2. Defining New Types -- 
http://docs.python.org/extending/newtypes.html. 

e Inthe Tools/framer directory of the Python source code distribution there is an 
application that will generate a skeleton for an extension type from a specification 
object written in Python. Run Tools/framer/example.py to see it in action. 

And, you can use Pyrex to generate a new built-in type. To do so, implement a 
Python/Pyrex class and declare the class with the Pyrex keyword cdef. In fact, you may 
want to use Pyrex to generate a minimal extension type, and then edit that generated code 
to insert and add functionality by hand. See the Pyrex section for an example. 


Pyrex also goes some way toward giving you access to (existing) C structs and functions 
from Python. 


2.5.8 Extension classes 
Extension classes the easy way -- SWIG shadow classes. 
Start with an implementation of a C++ class and its header file. 


Use the following SWIG flags: 


swig -ct+ -python mymodule.i 


More information is available with the SWIG documentation at: 
http://www.swig.org/Doc1.3/Python.html. 


Extension classes the Pyrex way -- An alternatie is to use Pyrex to compile a class 
definition that does not have the cdef keyword. Using cdef on the class tells Pyrex to 
generate an extension type instead of a class. You will have to determine whether you 
want an extension class or an extension type. 


2.6 Parsing 


Python is an excellent language for text analysis. 
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In some cases, simply splitting lines of text into words will be enough. In these cases use 
string.split(). 


In other cases, regular expressions may be able to do the parsing you need. If so, see the 
section on regular expressions in this document. 


However, in some cases, more complex analysis of input text is required. This section 
describes some of the ways that Python can help you with this complex parsing and 
analysis. 


2.6.1 Special purpose parsers 


There are a number of special purpose parsers which you will find in the Python standard 


library: 


ConfigParser parser - Configuration file parser -- 
http://docs.python.org/library/configparser.html 

getopt -- Parser for command line options -- 
http://docs.python.org/library/getopt.html 

optparse -- More powerful command line option parser -- 
http://docs.python.org/library/optparse.html 

urlparse -- Parse URLs into components -- 
http://docs.python.org/library/urlparse.html 

csv -- CSV (comma separated values) File Reading and Writing -- 
http://docs.python.org/library/csv.html#module-csv 
os.path - Common pathname manipulations -- 
http://docs.python.org/library/os.path.html 


XML parsers and XML tools -- There is lots of support for parsing and processing XML 
in Python. Here are a few places to look for support: 


The Python standard library -- Structured Markup Processing Tools -- 
http://docs.python.org/library/markup.html. 

In particular, you may be interested in xml.dom.minidom - Lightweight DOM 
implementation -- http://docs.python.org/library/xml.dom.minidom.html. 
ElementTree -- You can think of ElementTree as an enhanced DOM (document 
object model). Many find it easier to use than minidom. ElementTree is in the 
Python standard library, and documentation is here: ElementTree Overview -- 
http://effbot.org/zone/element-index.htm. 

Lxml mimics the ElementTree API, but has additional capabilities. Find out about 
Lxml at lxml -- http://codespeak.net/Ixml/index.html -- Note that xml also has 
support for XPath and XSLT. 

Dave's support for Python and XML -- http://www.rexx.com/~dkuhlman. 
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2.6.2 Writing a recursive descent parser by hand 
For simple grammars, this is not so hard. 


You will need to implement: 


e A recognizer method or function for each production rule in your grammar. Each 


recognizer method begins looking at the current token, then consumes as many 
tokens as needed to recognize it's own production rule. It calls the recognizer 
functions for any non-terminals on its right-hand side. 


e A tokenizer -- Something that will enable each recognizer function to get tokens, 


one by one. There are a variety of ways to do this, e.g. (1) a function that 


produces a list of tokens from which recognizers can pop tokens; (2) a generator 


whose next method returns the next token; etc. 
As an example, we'll implement a recursive descent parser written in Python for the 
following grammer: 


Prog s3— Command |||) command) Piaog 

Command ::= Func_call 

TUinCaac aie e se — 2 here: a () eimetenctcml elaine ee 

in Ghoie eel Wists So) inibhael oe lil || Iaibhareeelik UY lathe reyebl abakisie 
Term = <word> 


Here is an implementation of a recursive descent parser for the above grammar: 


#!/usr/bin/env python 


ww 


A recursive descent parser example. 


Wiscicie: 

python rparser.py [options] <inputfile> 
Options: 

=I, ——Aniedhye Display this help message. 
Example: 


python rparser.py myfile.txt 


The> oi ainmeas: 


Prog ::= Command | Command Prog 

Command ::= Func_call 

inilayese@enlil BS ikea VY inthe era WL hale) 

inGloe Celik Uilsieke So) ibis eel || iia ceil V4 laine! ieleuL I= okey 


Term = <word> 


ww 


sijeoias Shs! 
IMNVOrLE String, 
import types 
import getopt 
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To use the IPython interactive shell to inspect your running 
application, uncomment the following lines: 


# from IPython.Shell import IPShellEmbed 
# ipshell = IPShellEmbed((), 
# banner = '>>>>>>>> Into IPython >>>>>>>>', 

# exit_msg = '<<<<<<<< Out of IPython <<<<<<<<"') 


Then add the following line at the point in your code where 
you want to inspect run-time values: 


ipshell('some message to identify where we are') 


For more information see: http://ipython.scipy.org/moin/ 


se COs HH HRC CO HE Oo ECs #H@sowSE Os SEC =HHCOs=E CO OE OE OE 


# Constants 


# AST node types 
NoneNodeType = 0 

Pic Oeineceiywies = I 
CommandNodeType = 2 
FuncCallNodeType = 3 
FuncCallListNodeType = 4 
TermNodeType = 5 


# Token types 
NoneTokType = 
LParTokType = 
RE acl Okny pe 
WordTokType = 
CommalTokType = 
HORROKTYy OS <= <5 


ll 
CIS) 


4 


# Dictionary to map node type values to node type names 
NodeTypeDict = { 

NoneNodeType: 'NoneNodeType', 

ProgNodeType: 'ProgNodeType', 

CommandNodeType: 'CommandNodeType', 
FuncCallNodeType: 'FuncCallNodeType', 
FuncCallListNodeType: 'FuncCallListNodeType', 
TermNodeType: 'TermNodeType', 

} 


# 
# Representation of a node in the AST (abstract syntax tree). 
# 
Glless NS WiNocle + 
def __init__(self, nodeType, *args): 
self.nodeType = nodeType 
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self.children = [] 
for item in args: 
self.children.append (item) 
def show(self, level): 
self.showLevel (level) 
print 'Node -- Type %s' % NodeTypeDict [self .nodeType] 
level += 1 
ineye @losl lel “akion eyeulie, elguil Ikehesiort 
if Simstanee (Chaka ASMNedes 
child. show (level) 
elif type(child) == types.ListType: 
oes acetal shioy lat els 
item. show (level) 


else: 
self.showLevel (level) 

joucatians “Cloakilelo > ielaal kel 

def showLevel(self, level): 
for idx in range(level): 


joneatione 4 er 
# 
# The recursive descent parser class. 
+ Contains the "recognizer" methods, which implement the grammar 
tf rules (above), one recognizer method for each production rule. 
# 


Gubsigis) IPieorele ciiewee & 
(lene aligatie  (Usieuse )) 
pass 


def parseFile(self, infileName): 
self.infileName = infileName 
self.tokens = None 
Solr. cOken typo. None Lok voc 
self.token = '! 


self.lineNo = -1 

self.infile = file(self.infileName, 'r"') 
self.tokens = genTokens(self.infile) 
(erayes 


self.tokenType, self.token, self.lineNo = 
self.tokens.next () 
except StopIteration: 
raise RuntimeError, ‘Empty file' 
result = self.prog_reco() 
self.infile.close() 
self.infile = None 
return result 


def parseStream(self, instream): 
self.tokens = genTokens(instream, '<instream>') 
trys 
self.tokenType, self.token, self.lineNo = 
self.tokens.next () 
except StopIteration: 
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raise RuntimeError, 
result = self.prog_reco() 
return result 


Empty file’ 


def prog_reco(self): 

commandList = [ 
while 1: 

result = self.command_reco() 

if not result: 

break 

commandList.append (result) 

return ASTNode (ProgNodeType, commandList) 


def command_reco(self): 


if self.tokenType == EOFTokType: 
return None 
result = self.func_call_reco() 


return ASTNode (CommandNodeType, result) 


Glene seubee eel IL _seSie'@ ((fsvelie)) ¢ 
if self.tokenType == WordTokType: 
Gere AS MNeode (MenmNodelyoc, sc liane Olecim) 
self.tokenType, self.token, self.lineNo = 
Selle. cOc<eumls -inerac (()) 
if self.tokenType == LParTokType: 
self.tokenType, self.token, self.lineNo = 
SEINe eo) Sins iovesce (()) 


result = self.func_call_list_reco() 
it ee stale. 
if self.tokenType == RParTokType: 


self.tokenType, self.token, self.lineNo = \ 
self.tokens.next () 


Bebinn AST Node (HunceallliNodeivise,. ecrm, 
Osi) 
else: 
raise ParseError(self.lineNo, ‘missing right 
paren') 
elses 
raise ParseError(self.lineNo, ‘bad func call 
age!) 


eILSiSe 
raise ParseError(self.lineNo, 'missing left paren") 


else: 
return None 


det funcuccqlinstamecortselin i 
terms = [] 
while 1: 

result = self.func_call_reco() 
ist Wa@ce eee Siudleter 
break 
terms.append (result) 
if self.tokenType != CommaTokType: 
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break 
self.tokenType, self.token, self.lineNo = 
Selle cOleemls ney (()) 
return ASTNode (FuncCallListNodeType, terms) 


# 
# The parse error exception class. 
# 
class ParseError (Exception): 
def _ init__(self, lineNo, msg): 
RuntimeError.__init__(self, msg) 
self.lineNo = lineNo 
Sele msc = ims 
def getLineNo(self): 
return self.lineNo 
def getMsg(self): 
return self.msg 


def is_word(token): 
for letter in token: 
if letter not in string.ascii_letters: 
return None 
return 1 


# 

# Generate the tokens. 

# Usage: 

# gen = genTokens (infile) 

+ tokType, tok, lineNo = gen.next() 
# 


def genTokens(infile): 


( 
lineNo = 0 


while 1: 
lineNo += 1 
ieee: 
line = infile.next () 
ENCES 
yield (EOFTokType, None, lineNo) 
toks = line.split () 


hOE tOk dm tokis: 


Liv Seweme(eols)i: 

tokType = WordTokType 
elif tok == '(': 

tokiype — LParlokiype 
elif tok == ')': 

tokiype — RParlokiype 
elif tok == ',': 

tokType = CommaTokType 
yield (tokType, tok, lineNo) 


def test (infileName): 
parser = ProgParser () 
@joysioraiLIL(Y (GeScie)) sail VoGieell Ip) ice; ean )) 
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result = None 

Wives 
result = parser.parseFile(infileName) 

CxeCepe Haksehr wom, ovo: 
sys.stderr.write('ParseError: (%d) %s\n' % \ 

(exp.getLineNo(), exp.getMsg()) ) 

it result: 

result.show (0) 


def usage(): 
joes clo __ 
sys.exit (1) 


(lene invelaiol ()) 6 
args = sys.argv[l1:] 
ery: 
opts, args = getopt.getopt(args, 'h', ['help']) 
SXESIOL 5 
usage () 
relink = 1 
On Opt, Vall an opes: 
sae yore oy (la —ovelo! ) 
usage () 
if len(args) != 
usage () 
inputfile = args[0] 
test (inputfile) 


ee 


blest name == '  main__': 
#import pdb; pdb.set_trace() 
main () 


Comments and explanation: 


e The tokenizer is a Python generator. It returns a Python generator that can 
produce "(tokType, tok, lineNo)" tuples. Our tokenizer is so simple-minded that 
we have to separate all of our tokens with whitespace. (A little later, we'll see how 
to use Plex to overcome this limitation.) 

e The parser class (ProgParser) contains the recognizer methods that implement the 
production rules. Each of these methods recognizes a syntactic construct defined 
by arule. In our example, these methods have names that end with "_reco". 

e We could have, alternatively, implemented our recognizers as global functions, 
instead of as methods in a class. However, using a class gives us a place to "hang" 
the variables that are needed across methods and saves us from having to use 
("evil") global variables. 

e A recognizer method recognizes terminals (syntactic elements on the right-hand 
side of the grammar rule for which there is no grammar rule) by (1) checking the 
token type and the token value, and then (2) calling the tokenizer to get the next 
token (because it has consumed a token). 
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e A recognizer method checks for and processes a non-terminal (syntactic elements 
on the right-hand side for which there is a grammar rule) by calling the recognizer 
method that implements that non-terminal. 

e Ifarecognizer method finds a syntax error, it raises an exception of class 
ParserError. 

e Since our example recursive descent parser creates an AST (an abstract syntax 
tree), whenever a recognizer method successfully recognizes a syntactic construct, 
it creates an instance of class ASTNode to represent it and returns that instance to 
its caller. The instance of ASTNode has a node type and contains child nodes 
which were constructed by recognizer methods called by this one (i.e. that 
represent non-terminals on the right-hand side of a grammar rule). 

e Each time a recognizer method "consumes a token", it calls the tokenizer to get 
the next token (and token type and line number). 

e The tokenizer returns a token type in addition to the token value. It also returns a 
line number for error reporting. 

e The syntax tree is constructed from instances of class ASTNode. 

e The ASTNode class has a show method, which walks the AST and produces 
output. You can imagine that a similar method could do code generation. And, 
you should consider the possibility of writing analogous tree walk methods that 
perform tasks such as optimization, annotation of the AST, etc. 

And, here is a sample of the data we can apply this parser to: 


aaa ( ) 
isles) ( eer ( ) }} 
oléicl ( GSS ( )) 7 ux ( Gee ( ) 7 ialols ( ) 7 aaa ( ) ) 


And, if we run the parser on the this input data, we see: 


S$ python workbook045.py workbook045.data 
Node ==> lype PwogNodelype 
Node —-— Type CommandNodetype 
Node —— Ivype FuneealiNodetype 
Nod Masts: ikeuemiNexelelly4o) 
Chiila= sada 
Node -- Type FuncCallListNodeType 
Node -- Type CommandNodeType 
Node, =— yee Bunce alllinodeLype 
Nees -= Iyiss Ueuanioclelyiee 
(Claakikele lslele 
Node -- Type FuncCallListNodeType 
Node e=— Lype FuncealNodeivpe 
Node ==" lype BexmNodeLype 
Chamloks iece 
Node -- Type FuncCallListNodeType 
Node -- Type CommandNodeType 
Nock S— yas 1s bine Ge LIN oveleyjexSs 
Node == Type TermNodelype 
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Chaskel: ddd 
Node -- Type FuncCallListNodeType 
INheyekes = Ihjevs: yibualierC ell LINorele yee 
Nodes == type LexmNodeLype 
Child: eee 
Node -- Type FuncCallListNodeType 
Nels == iwi Bibine Cel LiNoce Tyce 
INoyeles == I\yjsrs WeseinNorcleEy ies 
Ghaklok: sett 
Node -- Type FuncCallListNodeType 
Node == Ivyee FuncealliiNodetfype 
Necks -= Iwi Terence lyiee 
(inal ikels “reife 
Node -- Type FuncCallListNodeType 
Node == lvoe Hune€ allliNocemyjoc 
Node == Type LermNodelype 
(Gleick ele: levouls 
Node -- Type FuncCallListNodeType 
NeYels == Iwies Biblia Ceill incl iyiee 
Nod Type TermNodeTyp 
Girailoks;  abaca 
Node -- Type FuncCallListNodeType 


2.6.3 Creating a lexer/tokenizer with Plex 


Lexical analysis -- The tokenizer in our recursive descent parser example was (for 
demonstration purposes) overly simple. You can always write more complex tokenizers 
by hand. However, for more complex (and real) tokenizers, you may want to use a tool to 
build your tokenizer. 


In this section we'll describe Plex and use it to produce a tokenizer for our recursive 
descent parser. 


You can obtain Plex at http://www.cosc.canterbury.ac.nz/~greg/python/Plex/. 
In order to use it, you may want to add Plex-1.1.4/Plex to your PYTHONPATH. 


Here is a simple example from the Plex tutorial: 


#!/usr/bin/env python 


ww 


Sample Plex lexer 


Usage: 
python plex_example.py inputfile 


ww 


sijeioiais, Shs 
import Plex 
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Clene Croblate alas) (sie eununieie,  uesme ye 
scanner.line_ count += 1 
jonerlioge es 6y10) 


def test (infileName): 


letter = Plex.Range ("AZaz") 
digit = Plex.Range("09") 
name = letter + Plex.Rep(letter | digit) 
number = Plex.Repl (digit) 
space = Plex.Any(" \t") 
einclisiae = Riles. Sicie (! \ia" )) 
#comment = Plex.Str('"') + Plex.Rep( Plex.AnyBut('"')) + 
Piles. Siete t)) 
GeESworoe—  PAkexeSitrm ("as Vehentt. Welksets Wena) 
lexicon = Plex.Lexicon([ 
(endline, Counts lamnes)i; 
(resword, "keyword'), 
(name, Yaeleimic.! }) 
(number, Vesiah a) Nr 
( Plexe Any (U4 /—< “ODE Bator) 
(space, Plex.IGNORE), 
# (Comment, "comment! ), 
(Plex SEC, iljona 7 
(Pilex eS bie (CO) jie Uneevsue ld) 
# comments surrounded by (* and *) 
(Pikes Sie (UIA Plex.Begin('comment')), 
Plex.State('comment', 
(Piles See (is) a Pilex. Beg imal’ iy 
(Plex.AnyChar, Plex.IGNORE), 


I), 
]) 


infile = open(infileName, "r") 
scanner = Plex.Scanner(lexicon, infile, infileName) 
scanner.line_count = 0 
while True: 

token = scanner.read() 

if token[0] is None: 

break 
position = scanner.position() 
posserm — Kiedy oc) ec ioosaie sora daly 
OOSaiesom|,2 I; )))) cll apwsic (ile) 
wokstrn — os) eoken (| 


Goksite — eokstra. Maisie (210) 
primk, “ss tok: cs EOkTypes 2S" 3 (posstr, tokstr, token 0] 
print 'line_count: %d' % scanner.line_count 


def usage(): 
joneakiona a ickore 
sys.exit (1) 


def main(): 
Bice — sys ecuatene|| Is) 


r) 
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seis 


if len(args) != 1: 
usage () 
infileName = args[0] 


test (infileName) 


name == iqstaiay VE 
#import pdb; pdb.set_trace() 
main () 


Here is a bit of data on which we can use the above lexer: 


mass = (height * (* some comment *) width * depth) / density 
totalmass = totalmass + mass 


And, when we apply the above test program to this data, here is what we see: 


S python plex_example.py plex_example.data 

(Gre 0) tok; "mass" tokType: ident 
(ibe 3) Goke = US" tokType: operator 
(ie 7) IGOL Gra ye (Cy tokType: lpar 

(ia 8) tok: Rhevghe" tokType: ident 
(le) ILS) RON oem aa tokType: operator 
Cl 36) toms “wali tokType: ident 
(1, 42) ok Wa tokType: operator 
(1, 44) tok “depen” tokType: ident 
(ie US) tok Wa tokType: rpar 

(Glee lg) ole. atau tokType: operator 
(lp 5)s)) tok: "density" tokType: ident 
(Ap 0) OM eo cllmasicn tokType: ident 
(2, 10) £Ok 2 t=" tokType: operator 
(Gin 2) bok “covalmass” tokType: ident 
Gre 22) okies Ma" tokType: operator 
(2, 24) tok: "mass" tokType: ident 


Larne: eons 2 


Comments and explanation: 


Create a lexicon from scanning patterns. 

See the Plex tutorial and reference (and below) for more information on how to 
construct the patterns that match various tokens. 

Create a scanner with a lexicon, an input file, and an input file name. 

The call "scanner.read()" gets the next token. It returns a tuple containing (1) the 
token value and (2) the token type. 

The call "scanner.position()" gets the position of the current token. It returns a 
tuple containing (1) the input file name, (2) the line number, and (3) the column 
number. 

We can execute a method when a given token is found by specifying the function 
as the token action. In our example, the function is count_lines. Maintaining a line 


Page 133 


A Python Book 


count is actually unneeded, since the position gives us this information. However, 
notice how we are able to maintain a value (in our case 1ine_count) as an 
attribute of the scanner. 

And, here are some comments on constructing the patterns used in a lexicon: 


Plex.Range constructs a pattern that matches any character in the range. 
Plex.Rep constructs a pattern that matches a sequence of zero or more items. 
Plex.Rep1 constructs a pattern that matches a sequence of one or more items. 
patl + pat2 constructs a pattern that matches a sequence containing pat 1 
followed by pat2. 

e patil | pat2 constructs a pattern that matches either pat1 or pat2. 

e Plex.Any constructs a pattern that matches any one character in its argument. 
Now let's revisit our recursive descent parser, this time with a tokenizer built with Plex. 
The tokenizer is trivial, but will serve as an example of how to hook it into a parser: 


#!/usr/bin/env python 


ww 


A recursive descent parser example using Plex. 
This example uses Plex to implement a tokenizer. 


Usage: 

python python_201_rparser_plex.py [options] <inputfile> 
Options: 

Sy eda Display this help message. 
Example: 


python python_201_rparser_plex.py myfile.txt 


The omammea es 


Prog ::= Command | Command Prog 

Command ::= Func_call 

iniloves @rsiLal “Be abe VM inhuborel ela ihilse  )) 

inybhovel eleulil Siksbiske 2 BR lyuhave! ersulil || leitiovel erie 27 9 laibhovesielamlIL a ibakene 


Term = <word> 


ww 


import sys, string, types 
import getopt 
import Plex 


## from IPython.Shell import IPShellEmbed 


## ipshell = IPShellEmbed((), 
Ht banner = '>>>>>>>> Into IPython >>>>>>>>', 
Ht exit_msg = '<<<<<<<< Out of IPython <<<<<<<<"') 


# Constants 
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# AST node t 
NoneNodeType 
ProgNodeType 
CommandNodeT 
FuncCallNode 
Func@alilivst 
TermNodeType 


# Token type 


LParTokType 
RParTokType 
WordTokType 
CommaTokType 
EOF TokType 


NodeTypeDict 
NoneNode 
ProgNode 
CommandN 
FuncCal 
FuncCal 


NoneTokType = 


ypes 
= 0 
= il 
ype = 2 
Type = 3 
NodeType = 4 
= 5) 
s 
= 0 
= 1 
= 2 
= 3 
= 4 
5) 


= 
ivipes 
Type: 
odeType: 


lNodeType: 
lListNodeType: 


TermNode 


} 
i 


# Representa 
# 
class ASTNod 
def __in 
self 

self 

IDOE 


def show 
self 
joneaLsal 
leve 
BONE 


def show 


A VASSE 


(CAL@lal he el 


node in the AST 


# Dictionary to map node type values to node type names 


"NoneNodeType', 
"ProgNodeType', 
"CommandNodeType', 
"FuncCallNodeType', 
"FuncCallListNodeType', 
 ReemNOdeiy oS ay 


(abstract syntax tree). 


e: 

it__(self, nodeType, *args): 
MOC IVS = ineycle lyjoxS 

wel leousetan —F [4] 

item in args: 
self.children.append (item) 
(self, level): 

. showLevel (level) 

t 'Node -- Type %s' % NodeTypeDict [self.nodeType] 
It ap dh 

Ghailid an sel ocha lide nk 


if MSN Stanee: (Glamis AS MNede)) s 


child.show(leve 

lif type (child) 
ioe alee ahiay «elaval [eke 

item. show (level) 


(S) 


lse: 


self.showLevel (1 
Wi @laiik lve l aye 
level): 


joneabigne 
Level (self, 


i) 
types.ListType: 


evel) 


Glatt lel 


for idx in range(level): 


joneaLinig  Y WF 


Page 135 


A Python Book 


# 

# The recursive descent parser class. 

# Contains the "recognizer" methods, which implement the grammar 
# rules (above), one recognizer method for each production rule. 
# 


Gubslais Ieee cietseue 6 
lene hier (svexlbie)) © 

self.tokens = None 
selt  eoken typo — Nonelokiyes 
self.token = '! 
self.lineNo = -1 
self.infile = None 
self.tokens 


ll 
Z 
(0) 
I} 
0) 


def parseFile(self, infileName): 
self.tokens = None 
Sele cO<emlyoes = Nomelel< Lye 
self.token = '' 
self.lineNo = -1 
self.infile feke(intihkeName, ii ) 
self.tokens = genTokens(self.infile, infileName) 
(GENES 


self.tokenType, self.token, self.lineNo = 
Selle. cOkems -inexac () 
except StopIteration: 
raise RuntimeError, ‘Empty file' 
result = self.prog_reco() 
self.infile.close() 
self.infile = None 
return result 


def parseStream(self, instream): 
self.tokens = None 
Sele cOl<emlyioSs = Nomeiel< byes 
Sell scoken = 


Selle laineNne = Si! 
self.tokens = genTokens(self.instream, '<stream>') 
jetayes 


self.tokenType, self.token, self.lineNo = 
Selle cOMems .inexxc (()) 
except StopIteration: 
raise RuntimeError, 'Empty stream' 
result = self.prog_reco() 
self.infile.close() 
self.infile = None 
eSiewhein wesbllic 


def prog_reco(self): 
commandList = [ 
while 1: 
result = self.command_reco() 
if net reise: 
break 
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commandList.append (result) 
return ASTNode (ProgNodeType, commandList) 


def command_reco(self): 


if self.tokenType == EOFTokType: 
return None 
result = self.func_call_reco() 


return ASTNode (CommandNodeType, result) 


Clare wie _eelliL icSie'o ((sieulig)) ¢ 
if self.tokenType == WordTokType: 
term = ASTNode(TermNodeType, self.token) 
self.tokenType, self.token, self.lineNo = 
Selle. cOcems inerxac () 
if self.tokenType == LParTokType: 
self.tokenType, self.token, self.lineNo = 
Selle, cO<ems -inerac () 


result = self.func_call_list_reco() 
aie wesc 
if self.tokenType == RParTokType: 


self.tokenType, self.token, self.lineNo = \ 
self.tokens.next () 


Gecinn A olMNode(HumeeomlNogdeivipe,  ecrnl, 
eS SisLc)) 
elses 
raise ParseError(self.lineNo, ‘missing right 
paren') 
else: 
raise ParseError(self.lineNo, ‘bad func call 
Ibsesiic ) 


SILSISs 
raise ParseError(self.lineNo, ‘missing left paren") 


else: 
return None 


def func_call_list_reco(self): 
terms = [] 
while 1: 
result = self.func_call_reco() 
if not result: 
break 
terms.append (result) 
if self.tokenType != CommaTokType: 
break 
self.tokenType, self.token, self.lineNo = 
self.tokens.next () 
return ASTNode (FuncCallListNodeType, terms) 


# 
# The parse error exception class. 
# 
class ParseError (Exception): 

def _ init__(self, lineNo, msg): 
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RuntimeError.__init__(self, msg) 
self.lineNo = lineNo 
Selle mse, = ims 
def getLineNo(self): 
return self.lineNo 
def getMsg(self): 
return self.msg 


# 

# Generate the tokens. 

# Usage - example 

# gen = genTokens (infile) 

# tokType, tok, lineNo = gen.next() 
# 


def genTokens(infile, infileName) : 


letter = Plex.Range ("AZaz") 

digit = Plex.Range("09") 

name = letter + Plex.Rep(letter | digit) 
jee = Ie beo< . Scie (1) 


rpar = Plex.Str(')') 
Comma —) Pllesey Ger (i, ) 
comment 
space = Plex.Any(" \t\n") 
lexicon Piles. Lexicon (| 
(name, 'word'), 
Ij S¥elae ¥ Ijeyee  )) 


i i 


( , 
(ia clize, V teieyeue! )) 5 
(comma, Commans)i, 
( 

( 


comment, Plex.IGNORE), 
space, Plex.IGNORE), 


]) 


Plex.Str("#") + Plex.Rep (Plex.AnyBut ("\n") ) 


scanner = Plex.Scanner(lexicon, infile, 


infileName) 


while 1: 
tokenType, token = scanner.read() 


name, lineNo, columnNo = scanner.position() 


if tokenType == None: 
Gokivyoe — HOP LOKType 
token — None 
elif tokenType == 'word': 
Olio = Worecll@lelby ise 
elif tokenType == 'lpar': 
COM: = Ieee We) IyioxS 
elif tokenType == 'rpar': 
Pokhyp ce RE ciahokmyiac 
elif tokenType == 'comma': 
tokiype = Commalokiype 
elses 
tokiype — NoneTokiype 
Gok > reken 
yield (tokType, tok, lineNo) 


def test (infileName): 
Dalwsery—) Prog Paieserm |) 
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#ipshell(' (test) #1\nCtrl-D to exit') 


esl = Ie 

try. 
result = parser.parseFile(infileName) 

Grxcejeie Je clicselhieis@ne,, S>qeie 
sys.stderr.write('ParseError: (%d) %s\n' % \ 


(exp.getLineNo(), exp.getMsg())) 
it result: 
result.show (0) 


def usage(): 
joneakionm clove 


sys.exit (-1) 


def main(): 


args = Ssvis ano |e] 
Gry: 
ODES, -aALds — Cevope CGerope (ands, » “ny i belo ug) 
SXCSIONL € 
usage () 
Om Opt, voll in Opes: 
Hae OOS JON (lay neo) he 
usage () 
if len(args) != 1: 
usage () 
infileName = args[0] 


test (infileName) 


siesta name == '  main__': 
#import pdb; pdb.set_trace() 
main() 


And, here is a sample of the data we can apply this parser to: 


# Test for recursive descent parser and Plex. 
# Command #1 


aaa () 
# Command #2 
bbb (cece ()) # An end of line comment. 


# Command #3 
oleic (Se) wits (leyeiej (7 lalla (77 abatal ()) ))) 
# End of test 


And, when we run our parser, it produces the following: 


S$ python plex_recusive.py plex_recusive.data 
Node -- Type ProgNodeType 
Node -- Type CommandNodeType 
Node -- Type FuncCallNodeType 
Node -- Type TermNodeType 
(Closlikele vale 
Node -- Type FuncCallListNodeType 
Node -- Type CommandNodeType 
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Node —— Type Funcealilnodetype 
Neyels —= Iie Weueniioclelyiee 
Child: bbb 
Node -- Type FuncCallListNodeType 
Weeks >= Ibyasis: 13 Linea LIUNoveleyjexS 
Neodes==) lype lormNocely ac 
Chie micce 
Node -- Type FuncCallListNodeType 
Node -- Type CommandNodeType 
Nock —— yas I ulna UN ocleIyjoxS 
Nodes == lype, LexmNodeLype 
[Elaigtiliel tole! 
Node -- Type FuncCallListNodeType 
Nees == iiss Hibine Cell LiNccle Tyee 
Nod lype LlermNodelypeS 
Child: eee 
Node -- Type FuncCallListNodeType 
INloyekes = Ijovsy la blalele cL LIN@rele jee 
Nees == wise WeraniocleLyiee 
(SloiaLARGlg aeseAe 
Node -- Type FuncCallListNodeType 
Node t= ivoe une alMiNocemyjoc 
INoyeles = Isis escniNorcle lies 
GleiaLIkele “rere, 
Node -- Type FuncCallListNodeType 
Node —-— Type Fune@allNodetype 
Node == siype wcrmNocdelyle 
Child: hhh 
Node -- Type FuncCallListNodeType 
Node —s Lyoce HumneecumlN OCS 
Nodes == lypes LeonmNocelywos 
(loa Jbelg — alabat 
Node -- Type FuncCallListNodeType 


Comments: 


e Wecan now put comments in our input, and they will be ignored. Comments 
begin with a "#" and continue to the end of line. See the definition of comment in 
function genTokens. 

e This tokenizer does not require us to separate tokens with whitespace as did the 
simple tokenizer in the earlier version of our recursive descent parser. 

e The changes we made over the earlier version were to: 

1. Import Plex. 

2. Replace the definition of the tokenizer function genTokens. 

3. Change the call to genTokens so that the call passes in the file name, which is 
needed to create the scanner. 

e Our new version of genTokens does the following: 

1. Create patterns for scanning. 
2. Create a lexicon (an instance of Plex.Lexicon), which uses the patterns. 
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3. Create a scanner (an instance of Plex.Scanner), which uses the lexicon. 
4. Execute a loop that reads tokens (from the scanner) and "yields" each one. 


2.6.4 A survey of existing tools 
For complex parsing tasks, you may want to consider the following tools: 


e kwParsing -- A parser generator in Python -- 
http://gadfly.sourceforge.net/kw Parsing. html 

e PLY -- Python Lex-Yacc -- http://systems.cs.uchicago.edu/ply/ 

e PyLR -- Fast LR parsing in python -- 
http://starship.python.net/crew/scott/PyLR.html 

e Yapps -- The Yapps Parser Generator System -- 
http://theory.stanford.edu/~amitp/Y apps/ 

And, for lexical analysis, you may also want to look here: 


e Using Regular Expressions for Lexical Analysis -- 
http://effbot.org/zone/xml-scanner.htm 
e Plex -- http://www.cosc.canterbury.ac.nz/~greg/python/Plex/. 
In the sections below, we give examples and notes about the use of PLY and pyparsing. 


2.6.5 Creating a parser with PLY 
In this section we will show how to implement our parser example with PLY. 


First down-load PLY. It is available here: PLY (Python Lex-Yacc) -- 
http://www.dabeaz.com/ply/ 


Then add the PLY directory to your PY THONPATH. 


Learn how to construct lexers and parsers with PLY by reading doc/ply.html in the 
distribution of PLY and by looking at the examples in the distribution. 


For those of you who want a more complex example, see A Python Parser for the 
RELAX NG Compact Syntax, which is implemented with PLY. 


Now, here is our example parser. Comments and explanations are below: 


#!/usr/bin/env python 


ww 


AS pawser sescampiice 
This example uses PLY to implement a lexer and parser. 


Ailsve) epee: 


PEOG ss; — Command* 
Command ::= Func_call 
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Auhae eel eS keen YC Y inibiake eeu bavene 4 
Funcucaliwivst 22=" Fumewealil* 
Term = <word> 


Here is a sample "program" to use as input: 


# Test for recursive descent parser and Plex. 
# Command #1 


aaa () 
# Command #2 
lolly ((ereres()) )} + Nol Giorel ne laine: Croumiimeione 


# Command #3 
Cleloleren())y, seseae (lerefe (i, tallavant() 77 alalab())) ) 
# End of test 


ww 


import sys 

import types 

import getopt 

import ply.lex as lex 
anjeewic jorlyoWweIee cis! selce 


4 
# Globals 
a 


startlinepos = 0 


4 
# Constants 


tt 


# AST node types 
NoneNodeType = 
PacoreiNieyelS I\jors = 
CommandNodeType = 
CommandListNodeType = 
Rune liiNodelyper — 
FuncCallListNodeType = 
TermNodeType = 


HANuPWNHE OC 


# Dictionary to map node type values to node type names 
NodeTypeDict = { 


NoneNodeType: INOMEANCCIS Lyjoe! 
ProgNodeType: "ProgNodeType', 
CommandNodeType: "CommandNodeType', 
CommandListNodeType: "CommandListNodeType', 
idibuake (Cs UILILIN(exolS Ib qevese HuneCaliliNodel ype 7 

IE DhAeIC UL INal Sie Nelo a “abbas euL aL SicIN@CIS Iyer, 
TermNodeType: "TermNodeType', 


} 
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# Representation of a node in the AST 


# 


class ASTNode: 


# 


def showLevel(self, level): 


def __init__(self, nodeType, *args): 


SEINE ineclelyjoSs = mock i yjcrSs 

self.children = [] 

Lom Teen in args: 
self.children.append (item) 


append(self, item): 


self.children.append (item) 
show(self, level): 
self.showLevel (level) 


fe) 


print 'Node -—- Type: %s' % NodeTypeDict [self.nodeType] 


level += 1 
hom Whale ane Seiki eicliaal@kaetas 


ie Asinstanee (chadd, —AStNede : 


child. show (level) 


elif type(child) == types.ListType: 


in@ie abieeiq —abiay lait els 
item. show(level) 


else: 
self.showLevel (level) 
joxedinnie | Viaulauless Ys clade 


for idx in range(level): 
jsiealinns —Y usr 


# Exception classes 


# 


class LexerError (Exception): 


class ParserError (Exception): 
def __init__(self, msg, lineno, columnno): 


def _ init__(self, msg, lineno, columnno): 


Sele Msc = ims 

self.lineno = lineno 
self.columnno = columnno 
show(self): 
sys.stderr.write('Lexer error 


(Set kimene,, “seit. coukumnmme;, 


SEIle isc; = iusto 
self.lineno = lineno 
self.columnno = columnno 


(abstract syntax tree). 


self.msg) ) 


def show(self): 


fo) fo) 


SySs-_stderr write ('Parser error (Sd, sa) ss\nt 3 \ 
(self.lineno, self.columnno, self.msg) ) 


# 
# Lexer specification 
# 
tokens = ( 

'NAME', 
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'LPAR', 'RPAR', 
'COMMA', 
) 


# Tokens 


(eA 
1 IRIE TAI, 
jE (COMIMI = 
1G INVA, 


eC 
Ned 

\ 

[ 


iia A Ail ie AN 0 Sali 


# Ignore whitespace 
Eatonene. = eet! 


# Ignore comments ('#' to end of line) 
def t_COMMENT (t): 

ey NG ae 

pass 


Gek Eimewlane (se): 
i aioe 
global startlinepos 
startlinepos = t.lexer.lexpos - 1 
Ge lWeMm@ c= tee culiulie  @ormate (ial!) 


def t_error(t): 
global startlinepos 
msg = "Illegal character '%Ss'" % (t.value[0]) 
columnno = t.lexer.lexpos - startlinepos 
raise LexerError(msg, t.lineno, columnno) 


# 
# Parser specification 
# 
GlEiE jojenctere (ie) s 

MpROG = .Commeandalast" 

t[0] = ASTNode (ProgNodeType, t[1]) 


def p_command_list_1(t): 
"command_list : command' 


t[0] = ASTNode (CommandListNodeType, t[1]) 


def p_command_list_2(t): 


"command_list : command_list command' 
t[1l] .append(t[2]) 
LON] = eb) 


def p_command(t): 
Veommang = fume seal" 
t[0] = ASTNode (CommandNodeType, t[1]) 


Coe jo feibhoe eel Ge) ¢ 
ShUncmccll ete crum hb AR RiP ARS 
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t[0O] = ASTNode (FuncCallNodeType, t[1]) 
Cem jom tumcac alli 2 (tee 
USeiine een Lk term LPAR func_call_list RPAR' 
t[0] = ASTNode(FuncCallNodeType, t[1], t[3]) 
det petuneme al IA Vsti (er 
acibligics Ceibik jhatsic ie hake! eeulbily 
t[0O] = ASTNode (FuncCallListNodeType, t[1]) 
Glee jorewlove Seah Walsic 2A (Cc) & 
Ye Uinieue aul S\te EUncG caliniiste COMMA funcseali 
t[1l] .append(t[3]) 
OID =e (Ab 
def p_term(t): 
"term NAME! 
t[0] = ASTNode (TermNodeType, t[1]) 
def p_error(t): 
global startlinepos 
MSc) = Wiswimleeb< eicieore vele Vagisil ss ie gel lLinKS 
columnno = t.lexer.lexpos - startlinepos 
raise ParserError(msg, t.lineno, columnno) 
# 
# Parse the input and display the AST (abstract syntax tree) 


# 
def parse (infileName) : 
startlinepos 0 
# Build the lexer 
lex.lex (debug=1) 
# Build the parser 
Wales eile) 
# Read the input 
infile 
COmecine 
infile.cl 
IENES 


int2le. read () 
ose () 


# Do 
resul 


the parse 

Ee Vide ee Oase 

# Display the AST 
result.show (0) 

exes Ine<eielthigieoie 
exp. show () 

Exes I enesisicd iio, 
exp. show () 


exp: 


exp 


USAGE_TEXT _ clo 

def usage(): 
print USAGE_ 
sys.exit (-1) 


T 


EXT 


file (infileName, 


lye) 


(content) 
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def main(): 

args 

ery 
OPiESs, 

EXCEL. 
usage () 

relink il 

ROL opt, valli an copes: 
ae (Oj abo ((Y la 

usage () 

if len(args) != 
usage () 

infileName args [0] 

parse (infileName) 


Shydsi cg eliareiy || 1h a] 


args = getopt.getopt(args, 'h', ['help']) 


.=—hednow yt 
lg 


sie name imivetakion © 
#import pdb; pdb.set_trace() 


main () 


Applying this parser to the following input: 


# Test for recursive descent parser and Plex. 
# Command #1 
aaa () 

# Command #2 
lolol) 1tefeke' ()))) 
# Command #3 
ddd(eee(), fff (ggg(), 
# End of test 


# An end of line comment. 


ieee () yp ab2b (0) ) 


produces the following output: 


Node =—) hype. erogNodelyps 
Node -- Type: CommandListNodel 
Node -- Type: CommandNodeType 
NOdee =) Lye. eine easmlNode 
Necks == iyi Ueiciil\ecks 
Value: aaa 
Node -- Type: CommandNodeType 
Nodes lype: BuneealiNodel 
Nod Typ TermNode!l 
Welles Islolo 

Node -- Type: 
Node -- Type: 
Nod 


[ype 


Type 
iype 


[Type 


Type 
[Type 
Typ 


FuncCallListNode 
FuncCallNode!l 
TermNod 
Value: ccc 


Node -- Type: 


CommandNodeType 


No 


de -- Type: Func 
Node -- Type: 
Values sddd 
Node -- Type: 
Node -- Ty 


CallNodeType 
TermNodeType 


FuncCallListNodeType 
pe: FuncCallNodeType 
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Node =—) lypc. lommNocdoiyps 
Wellwes eee 
Node -- Type: FuncCallNodeType 
Node == ype: lermNodelype 
Welle 163618 
Node -- Type: FuncCallListNodeType 
Neves: == “Isis e liblielGeuL inioycle Tyas 
Noyes) == Nyse WisieuMi\vercle yore 
Value: ggg 
Node w-— lype: HuncealaiNode types 
Node =— ype: TermNodelype 
Weulibve 8  Iovallal 
Neel) = I\ysvS3 imblneliGeul IINoclelyioS 
Node ives. LoamNodelves 
Value: iii 


Comments and explanation: 


e Creating the syntax tree -- Basically, each rule (1) recognizes a non-terminal, (2) 
creates a node (possibly using the values from the right-hand side of the rule), and 
(3) returns the node by setting the value of t[0]. A deviation from this is the 
processing of sequences, discussed below. 
e Sequences -- p_command_list_1 and p_command_list_1 show how to handle 
sequences of items. In this case: 
© p_command_list_1 recognizes a command and creates an instance of 
ASTNode with type CommandListNodeType and adds the command to it as a 
child, and 

© p_command_list_2 recognizes an additional command and adds it (as a child) 
to the instance of ASTNode that represents the list. 

e Distinguishing between different forms of the same rule -- In order to process 
alternatives to the same production rule differently, we use different functions 
with different implementations. For example, we use: 

o p_func_call_1 to recognize and process "func_call : term LPAR RPAR" (a 
function call without arguments), and 

o p_func_call_2 to recognize and process "func_call : term LPAR func_call_list 
RPAR" (a function call with arguments). 

e Reporting errors -- Our parser reports the first error and quits. We've done this by 
raising an exception when we find an error. We implement two exception classes: 
LexerError and ParserError. Implementing more than one exception class enables 
us to distinguish between different classes of errors (note the multiple except: 
clauses on the try: statement in function parse). And, we use an instance of the 
exception class as a container in order to "bubble up" information about the error 
(e.g. a message, a line number, and a column number). 


Page 147 


A Python Book 


2.6.6 Creating a parser with pyparsing 


pyparsing is a relatively new parsing package for Python. It was implemented and is 
supported by Paul McGuire and it shows promise. It appears especially easy to use and 
seems especially appropriate in particular for quick parsing tasks, although it has features 
that make some complex parsing tasks easy. It follows a very natural Python style for 
constructing parsers. 


Good documentation comes with the pyparsing distribution. See file 
HowToUseParsing.html. So, I won't try to repeat that here. What follows is an attempt to 
provide several quick examples to help you solve simple parsing tasks as quickly as 
possible. 


You will also want to look at the samples in the examples directory, which are very 
helpful. My examples below are fairly simple. You can see more of the ability of 
pyparsing to handle complex tasks in the examples. 


Where to get it - You can find pyparsing at: Pyparsing Wiki Home -- 
http://pyparsing.wikispaces.com/ 


How to install it - Put the pyparsing module somewhere on your PY THONPATH. 


And now, here are a few examples. 


2.6.6.1 Parsing comma-delimited lines 


Note: This example is for demonstration purposes only. If you really to need to parse 
comma delimited fields, you can probably do so much more easily with the CSV (comma 
separated values) module in the Python standard library. 


Here is a simple grammar for lines containing fields separated by commas: 


import sys 
from pyparsing import alphanums, ZeroOrMore, Word 


fieldDef = Word(alphanums) 
lineDef = fieldDef + ZeroOrMore("," + fieldDef) 


Cle rSesie() & 
Glee isy sneha retatenic (flea) 
if len(args) != 1: 


print 'usage: python pyparsing_testl.py <datafile.txt>' 
sys.exit (-1) 


infilename = sys.argv[1] 
infile = file(infilename, 'r') 
form lane an, amitales 
fields = lineDef.parseString (line) 


print fields 
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test () 


Here is some sample data: 


abcd, defg 
Daas 22 GS) Gis) 


And, when we run our parser on this data file, here is what we see: 


S python comma_parser.py samplel.data 
P Felsrercl ee selekae | 
[CsA ail era ee ean eRe en pe Mewar endy SISO) gl 


Notes and explanation: 


e Note how the grammar is constructed from normal Python calls to function and 
object/class constructors. I've constructed the parser in-line because my example 
is simple, but constructing the parser in a function or even a module might make 
sense for more complex grammars. pyparsing makes it easy to use these these 
different styles. 

e Use "+" to specify a sequence. In our example, a LineDef isa fieldDef 
followed by .... 

e Use ZeroOrMore to specify repetition. In our example, a lLineDef isa 
fieldDef followed by zero or more occurances of comma and fieldDef. 
There is also OneOrMore when you want to require at least one occurance. 

e Parsing comma delimited text happens so frequently that pyparsing provides a 
shortcut. Replace: 


lineDef = fieldDef + ZeroOrMore("," + fieldDef) 


with: 


lineDef = delimitedList (fieldDef) 


And note that delimitedList takes an optional argument delim used to specify 
the delimiter. The default is a comma. 


2.6.6.2 Parsing functors 


This example parses expressions of the form func(argl, arg2, arg3): 


from pyparsing import Word, alphas, alphanums, nums, ZeroOrMore, 
Literal 

lparen = Literal("(") 

rearen = Literal (™)") 

identifier = Word(alphas, alphanums + "_") 

integer = Word( nums ) 

functor = identifier 

arg = identifier | integer 
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args = arg + ZeroOrMore("," + arg) 

expression = functor + lparen + args + rparen 

Clee icesic () £ 
content = raw_input ("Enter an expression: ") 
parsedContent = expression.parseString (content) 
print parsedContent 

ices () 

Explanation: 


e Use Literal to specify a fixed string that is to be matched exactly. In our example, 
a lparen is a (. 

e Word takes an optional second argument. With a single (string) argument, it 
matches any contiguous word made up of characters in the string. With two 
(string) arguments it matches a word whose first character is in the first string and 
whose remaining characters are in the second string. So, our definition of 
identifier matches a word whose first character is an alpha and whose remaining 
characters are alpha-numerics or underscore. As another example, you can think 
of Word("0123456789") as analogous to a regexp containing the pattern "[0-9]+". 

e Use a vertical bar for alternation. In our example, an arg can be either an identifier 
or an integer. 


2.6.6.3 Parsing names, phone numbers, etc. 


This example parses expressions having the following form: 


Hayorbher neCuanikehe 
[name] [phone] ehiteyA, Siecle: vaal\6)|| 
Last, first 2 oS seu Cine, Ca 9999 


Here is the parser: 


import sys 
from pyparsing import alphas, nums, ZeroOrMore, Word, Group, 
Suppress, Combine 


lastname = Word(alphas) 

firstname = Word(alphas) 

city = Group (Word(alphas) + ZeroOrMore (Word (alphas) ) ) 
state = Word(alphas, exact=2) 

zip = Word(nums, exact=5) 


name = Group(lastname + Suppress(",") + firstname) 

phone = Combine (Word(nums, exact=3) + "-" + Word(nums, exact=3) + "-" 
+ Word(nums, exact=4) ) 

lLocatvon — iGeoupiictty + Suppress", )) t+ Stade -  z7 9) 

record = name + phone + location 
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Clee ces ()) & 
SCS ys yisa res Cys [ella s| 
if len(args) != 1: 


print ‘usage: python pyparsing_test3.py <datafile.txt>' 
sys.exit (-1) 


infilename = sys.argv[l1] 
iigueal ey == teal Ibe (ahiane dh eines, sie’ )} 
neee ILahiayey abiey abialie ak Ile\< 
line = line.strip() 
if dine and Hine) YS vats 
fields = record.parseString (line) 


print fields 


test () 


And, here is some sample input: 


Jabberer, Jerry W222 SiS) Bakers pital CA So ihiel 
Kackler, Kerry Wii 22 2Z= 3.534 Dieeisinoy, (iy Ss ill 
Louderdale, Larry Lid 222—=3393)5 Los Angeles, CA 94001 


Here is output from parsing the above input: 


[i wWweabbeser";) “derry "| 7. Silt 222-s333", |i Bakers tieia |, “eat, 
US Salads Ya 
i Karcher re Skerry) 222-2834 ii  meressnor ily. MEAN Y Oisdi 2 0a) a) 
[[*bouderdaile’ 7 “harny" |; tllil=202-=3335', |i hos", “Angeles ’ |), ‘eAt, 
VIACOM) || 

Comments: 


e We use the len=n argument to the Word constructor to restict the parser to 


accepting a specific number of characters, for example in the zip code and phone 
number. Word also accepts min=n'' and ~ ~max=n to enable you to restrict 


the length of a word to within a range. 

e We use Group to group the parsed results into sub-lists, for example in the 
definition of city and name. Group enables us to organize the parse results into 
simple parse trees. 


e We use Combine to join parsed results back into a single string. For example, in 


the phone number, we can require dashes and yet join the results back into a 
single string. 

e We use Suppress to remove unneeded sub-elements from parsed results. For 
example, we do not need the comma between last and first name. 


2.6.6.4 A more complex example 


This example (thanks to Paul McGuire) parses a more complex structure and produces a 


dictionary. 
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Here is the code: 


from pyparsing import Literal, Word, Group, Dict, 
nums, \ 

delimitedList 
Aeneas josseakiohe 


ZeroOrMore, alphas, 


datatable = heading + Dict( ZeroOrMore(rowData) ) 


def main(): 
# Now parse data and print results 
data = datatable.parseString(testData) 
pein Meleitzas") “dated 
joieding Weleiesls cisilaline ()) 347, 
PeLIne-porime(datanas lassie) ) 
print "data keys:", data.keys() 
joneakions “4Cletoret || Virationy |) 2 teleneet || Saalsaiy | 
print "data.max:", data.max 
site name == Maine": 
main() 


eSie Deluca —e a 
Al igsdl (euil Dial A2 B2 EZ D2 

min 7 43 a ks) 82 98 il Sy 
max dha BZ, LO ike 85 Ee 4 39 
ave 9 47 8 16 84 106 S| 38 
sdev il 3 il ub il 3 al i 

# Define grammar for datatable 

heading = (Literal ( 

+ 

i Al Bl Cl D1 A2 B2 EZ D2 

suppress () 

vert = Literal("|").suppress () 

number = Word(nums) 

rowData = Group( vert + Word(alphas) + vert + 

delimitedList (number,"|") + 

vert ) 

trailing = Literal ( 

suppress () 


ap eesti ILaliave} 


ae 


When we run this, it produces the following: 


|data: [eos ie See eg WOM yO An yn sous, 


ale es VB eh, 
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Wms Mle yy Loe) Oe. Ps “Meer. ae ee OL 

‘ave', Oe Be ee Yala MEG iat DOG ves eas eS Onell ee 

selene 5 pele Vay aes Male pelts uscialee ied Fer Vala a 

data asimrsite (jest (timate ue Ae Mako! eae Se Val Usa 
‘mee A Lit, YER, 10%, TET, 88st, "112", 340, Toary, 

‘ave', Wig Be ee eons VEG"; its Aa VANOe oes Vests allies 

sce Urlely ene unlit Us lee Usa valle ea 

data keys: ['ave', 'min', 'sdev', 'max'] 

detalii: Ea Sea Me os ee eyes al. east | 

datvarmmascss Vii 52477. Os yt Me ee Ue ee On| 


Notes: 


e Note the use of Dict to create a dictionary. The print statements show how to get 
at the items in the dictionary. 

e Note how we can also get the parse results as a list by using method asList. 

e Again, we use suppress to remove unneeded items from the parse results. 


2.7 GUI Applications 


2.7.1. Introduction 


This section will help you to put a GUI (graphical user interface) in your Python 
program. 


We will use a particular GUI library: PYGTK. We've chosen this because it is reasonably 
light-weight and our goal is to embed light-weight GUI interfaces in an (possibly) 
existing application. 


For simpler GUI needs, consider EasyGUI, which is also described below. 


For more heavy-weight GUI needs (for example, complete GUI applications), you may 
want to explore WxPython. See the WxPython home page at: http://www.wxpython.org/ 


2.7.2 PyGtk 
Information about PyGTK is here: The PyGTK home page -- http://www.pygtk.org//. 


2.7.2.1 A simple message dialog box 


In this section we explain how to pop up a simple dialog box from your Python 
application. 


To do this, do the following: 


1. Import gtk into your Python module. 
2. Define the dialog and its behavior. 
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3. Create an instance of the dialog. 
4. Run the event loop. 
Here is a sample that displays a message box: 


#!/usr/bin/env python 


aijeeraie Side! 
import getopt 
import gtk 


class MessageBox (gtk.Dialog) : 
def __init__(self, message="", buttons=(), pixmap=None, 
modal= True): 
gtk. Dialog. init (siel £) 
self.connect ("destroy", self.quit) 
self.connect ("delete_event", self.quit) 
it modal: 
self.set_modal (True) 
hbox = gtk.HBox(spacing=5) 
hbox.set_border_width (5) 
self.vbox.pack_start (hbox) 
hbox. show () 
if pixmap: 
self.realize() 
pixmap = Pixmap(self, pixmap) 
hbox.pack_start (pixmap, expand=False) 
pixmap.show() 
label = gtk.Label (message) 
hbox.pack_start (label) 
label.show() 
for text in buttons: 
b = gtk.Button (text) 
b.set_flags (gtk.CAN_DEFAULT) 
b.set_data("user_data", text) 
beconnece (elicked",), sselioclack) 
self.action_area.pack_start (b) 


b. show () 
self.ret = None 
def quit(self, *args): 

self.hide () 


self.destroy () 
gtk.main_quit () 
Clase @llivele((siSellic,  loibliecxenal)) 2 
self.ret = button.get_data("user_data") 
self.quit () 


# create a message box, and return which button was pressed 


def message_box(title="Message Box", message="", buttons=(), 
pixmap=None, 
modal= True): 


With, Seite tele Wel (eatelae)) 


win = MessageBox (message, buttons, pixmap=pixmap, modal=modal) 
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win.show() 
gtk.main() 
return win.ret 


def test (): 
result = message_box(title='Test #1', 
message='Here is your message', 
buttons=('Ok', 'Cancel')) 
joueabins)  Jieeciblies Vy sasreyullic 


USAGE _TExT = 7am 
Wisicicicg 
python simple_dialog.py [options] 
Options: 
= 19) lately Display this help message. 
Example: 


python simple_dialog.py 


ww 


def usage(): 
print USAGE_TEXT 
sys.exit (-1) 


(ClSie invelaLial ()) £ 


aes. = (SVis ice Oia less 
rye: 
opts, args = getopt.getopt(args, ‘h', ['help']) 
SMESIOE 5 
usage () 
relink = 1 
ROR Ope, Vall an opus: 
Shae yo OU (Malay = ioverllios jin 
usage () 
if len(args) != 0: 
usage () 
Gest) 
age name == ieatiny 5 
#import pdb; pdb.set_trace() 
main () 


Some explanation: 


e First, we import gtk 
e Next we define a class MessageBox that implements a message box. Here are a 
few important things to know about that class: 
o Itis asubclass of gtk.Dialog. 
o It creates a label and packs it into the dialog's client area. Note that a Dialog is 
a Window that contains a vbox at the top of and an action_area at the bottom 
of its client area. The intension is for us to pack miscellaneous widgets into 
the vbox and to put buttons such as "Ok", "Cancel", etc into the action_area. 
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o It creates one button for each button label passed to its constructor. The 
buttons are all connected to the click method. 

o The click method saves the value of the user_data for the button that was 
clicked. In our example, this value will be either "Ok" or "Cancel". 

e And, we define a function (message_box) that (1) creates an instance of the 
MessageBox class, (2) sets its title, (3) shows it, (4) starts its event loop so that it 
can get and process events from the user, and (5) returns the result to the caller (in 
this case "Ok" or "Cancel"). 

e Our testing function (test) calls function message_box and prints the result. 

e This looks like quite a bit of code, until you notice that the class MessageBox and 
the function message_box could be put it a utility module and reused. 


2.7.2.2 A simple text input dialog box 


And, here is an example that displays an text input dialog: 


#!/usr/bin/env python 


import sys 
import getopt 
import gtk 


class EntryDialog( gtk.Dialog): 
def __init__(self, message="", default_text='', modal=True): 
gtk.Dialog.__init__ (self) 
self.connect ("destroy", self.quit) 
self.connect ("delete_event", self.quit) 
if modal: 
self.set_modal (True) 
box = gtk.VBox (spacing=10) 
box.set_border_width(10) 
self.vbox.pack_start (box) 


box.show () 
if message: 
label = gtk.Label (message) 


box.pack_start (label) 
label.show () 

Selig Getoheisy, = Gils 5 voces () 
self.entry.set_text (default_text) 
lXOps OLE _Sieeec (SEINE Sie icy) 
self.entry.show() 
self.entry.grab_focus () 

button = gtk.Button ("OK") 

HUEECh COonnecE (clicked: = Selim aci mick) 
button.set_flags (gtk.CAN_DEFAULT) 
self.action_area.pack_start (button) 
button. show () 

button.grab_default () 

bUEton —soEek Bueron (iGancell™) 


Page 156 


A Python Book 


button.connect ("clicked", self.quit) 
button.set_flags (gtk.CAN_DEFAULT) 
self.action_area.pack_start (button) 
button. show () 


self.ret = None 
def quit(self, w=None, event=None): 
self .hide () 


self.destroy () 
gtk.main_quit () 
eck Click (sellt, Jouetom) 
self.ret = self.entry.get_text () 
self.quit () 


def input_box(title="Input Box", message="", default_text="', 
modal=True): 
win = EntryDialog(message, default_text, modal=modal) 
win.set_title (title) 
win. show () 
gtk.main () 
return win.ret 


def test(): 
result = input_box(title='Test #2', 
message='Enter a valuexxx:', 
default_text='a default value') 
ie Cesta Nome. 
print 'Canceled' 


else: 
DiseiMey Mae sues Nasu (so. msesuilte 

USAGE_TEXT = """ 
Usage: 

python simple_dialog.py [options] 
Options: 

=o, ——loeylkyo) Display this help message. 
Example: 


python simple_dialog.py 


ww 


def usage(): 
print USAGE_TEXT 
sys.exit (-1) 


(lene invelaliol ()) = 


aegis, = Sis creo, (lata 
wry: 

opts, args = getopt.getopt(args, ‘h', ['help']) 
excep. 

usage () 
relink = 1 
hoe Opt, Vici an epes: 

SEG OSE LOU (ela ee inverlLo Mie 

usage () 
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it léeni(args) Y= 0% 
usage () 
ices () 
aLae name == Mates 
#import pdb; pdb.set_trace() 
main () 
Most of the explanation for the message box example is relevant to this example, too. 


Here are some differences: 


e Our EntryDialog class constructor creates instance of gtk.Entry, sets its default 


value, an 
e Thecons 


d packs it into the client area. 
tructor also automatically creates two buttons: "OK" and "Cancel". The 


"OK" button is connect to the click method, which saves the value of the entry 
field. The "Cancel" button is connect to the quit method, which does not save the 


value. 


e And, if class EntryDialog and function input_box look usable and useful, add 


them to your utility gui module. 


2.7.2.3 A file selection dialog box 


This example shows a file selection dialog box: 


#!/usr/bin/env python 


import sys 
import getopt 
aeeais erel< 


class FileChooser (gtk.FileSelection): 
def __ init__(self, modal=True, multiple=True): 
gtk.FileSelection.__init__(self) 
self.multiple = multiple 
self.connect ("destroy", self.quit) 
self.connect ("delete_event", self.quit) 
if modal: 
self.set_modal (True) 
self.cancel_button.connect ('clicked', self.quit) 
self.ok_button.connect ('clicked', self.ok_cb) 
if multiple: 
self.set_select_multiple (True) 
self.ret = None 
def quit(self, *args): 
self.hide () 
self.destroy () 
Gek emer nec uates() 
Adem (okmicbi(seis ia) 
if self.multiple: 
self.ret = self.get_selections () 
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else: 
self.ret = self.get_filename () 
self.quit () 


def file_sel_box(title="Browse", modal=False, multiple=True): 
win = FileChooser(modal=modal, multiple=multiple) 
win.set_title (title) 
win. show () 
gtk.main () 
return win.ret 


def file_open_box (modal=True): 
return file_sel_box("Open", modal=modal, multiple=True) 
def file_save_box (modal=True): 
return file_sel_box("Save As", modal=modal, multiple=False) 


Clg wesc ()) - 
result = file_open_box() 
print ‘open result:', result 


result = file _save_box() 
print 'save result:', result 


USAGE_TEXT = """ 
USeces 
python simple_dialog.py [options] 
Options: 
=l0bp ni Ibo) Display this help message. 
Example: 


python simple_dialog.py 


ww 


def usage(): 
print USAGE_T 
sys.exit (-1) 


rea 
x 
il 


lene ineialian())) € 
aes: = SViswiak oie lis 
er ye 
opts, args = getopt.getopt(args, 'h', ['help']) 
SESS & 
usage () 
relink = 1 
RO Ope, valk in Opes: 
Shia Gjoie sigh (4 lot —loveulios jive 
usage () 
if len(args) != 
usage () 
ces () 


OF 


aie name == main 
main () 
#import pdb 
#pdb.run('main() ') 
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A little guidance: 


e There is a pre-defined file selection dialog. We sub-class it. 

e This example displays the file selection dialog twice: once with a title "Open" and 
once with a title "Save As". 

e Note how we can control whether the dialog allows multiple file selections. And, 
if we select the multiple selection mode, then we use get_selections instead of 
get_filename in order to get the selected file names. 

e The dialog contains buttons that enable the user to (1) create a new folder, (2) 
delete a file, and (3) rename a file. If you do not want the user to perform these 
operations, then call hide_fileop_buttons. This call is commented out in our 
sample code. 

Note that there are also predefined dialogs for font selection (FontSelectionDialog) and 
color selection (ColorSelectionDialog) 


2.7.3 EasyGul 


If your GUI needs are minimalist (maybe a pop-up dialog or two) and your application is 
imperative rather than event driven, then you may want to consider EasyGUI. As the 
name suggests, it is extremely easy to use. 


How to know when you might be able to use EasyGUI: 


e Your application does not need to run in a window containing menus and a menu 
bar. 

e Your GUI needs amount to little more than displaying a dialog now and then to 
get responses from the user. 

e You do not want to write an event driven application, that is, one in which your 
code sits and waits for the the user to initiate operation, for example, with menu 
items. 

EasyGUI plus documentation and examples are available at EasyGUI home page at 
SourceForge -- http://easygui.sourceforge.net/ 


EasyGUI provides functions for a variety of commonly needed dialog boxes, including: 


e A message box displays a message. 

A yes/no message box displays "Yes" and "No" buttons. 

A continue/cancel message box displays "Continue" and "Cancel" buttons. 

A choice box displays a selection list. 

An enter box allows entry of a line of text. 

An integer box allows entry of an interger. 

A multiple entry box allows entry into multiple fields. 

Code and text boxes support the display of text in monospaced or porportional 
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fonts. 
e File and directory boxes enable the user to select a file or a directory. 
See the documentation at the EasyGUI Web site for more features. 


For a demonstration of EasyGUI's capabilities, run the easygui.py as a Python script: 


S$ python easygui.py 


2.7.3.1 A simple EasyGUI example 


Here is a simple example that prompts the user for an entry, then shows the response in a 
message box: 


import easygui 


def testeasygui(): 

response = easygui.enterbox(msg='Enter your name:', title='Name 
Entry') 

easygui.msgbox (msg=response, title='Your Response') 


testeasygui () 


2.7.3.2 An EasyGUI file open dialog example 


This example presents a dialog to allow the user to select a file: 


import easygui 


Clg wSsic()) & 
response = easygui.fileopenbox (msg='Select a file') 
print 'file name: %s' % response 


test () 


2.8 Guidance on Packages and Modules 


2.8.1 Introduction 


Python has an excellent range of implementation organization structures. These range 
from statements and control structures (at a low level) through functions, methods, and 
classes (at an intermediate level) and modules and packages at an upper level. 


This section provides some guidance with the use of packages. In particular: 


e How to construct and implement them. 
e How to use them. 
e How to distribute and install them. 
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2.8.2 Implementing Packages 
A Python package is a collection of Python modules in a disk directory. 


In order to be able to import individual modules from a directory, the directory must 
contain a file named __init__.py. (Note that requirement does not apply to directories that 
are listed in PYTHONPATH.) The __init__.py serves several purposes: 


e The presence of the file __init__.py in a directory marks the directory as a Python 
package, which enables importing modules from the directory. 

e The first time an application imports any module from the directory/package, the 
code in the module __init__ is evaluated. 

e Ifthe package itself is imported (as opposed to an individual module within the 
directory/package), then it is the __init__ that is imported (and evaluated). 


2.8.3 Using Packages 


One simple way to enable the user to import and use a package is to instruct the use to 
import individual modules from the package. 


A second, slightly more advanced way to enable the user to import the package is to 
expose those features of the package in the __init__ module. Suppose that module mod1 
contains functions funla and funlb and suppose that module mod2 contains functions 
fun2a and fun2b. Then file ___ in it__. py might contain the following: 


from modl import funla, funlb 
Evom MOGZ Ampoce funZa, funmZ 


Then, if the following is evaluated in the user's code: 


import testpackages 


Then testpackages will contain funla, funlb, fun2a, and fun2b. 


For example, here is an interactive session that demostrates importing the package: 


>>> import testpackages 
Pee Owing elise (eSsioeie xaos) 


[Sb nntlithimse 28 -  edocL lls. ~ apie. = names =), 
Gweini’ 
ewiolay, “setiniloy, ~“iebiazeY, “atuiaio!, “see ~~ aavevel 2 Y 


2.8.4 Distributing and Installing Packages 


Distutils (Python Distribution Utilities) has special support for distrubuting and installing 
packages. Learn more here: Distributing Python Modules -- 
http://docs.python.org/distutils/index.html. 
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As our example, imagine that we have a directory containing the following: 


Testpackages 

Testpackages/README 
Testpackages/MANIFEST.in 
Testpackages/setup.py 
Testpackages/testpackages/__init__.py 
Testpackages/testpackages/modl.py 
Testpackages/testpackages/mod2.py 


Notice the sub-directory Testpackages/testpackages containing the file__ init__.py. 
This is the Python package that we will install. 


We'll describe how to configure the above files so that they can be packaged as a single 
distribution file and so that the Python package they contain can be installed as a package 
by Distutils. 


The MANIFEST. in file lists the files that we want included in our distribution. Here is 
the contents of our MANIFEST. in file: 


include README MANIFEST MANIFEST.in 
include setup.py 
include testpackages/*.py 


The setup.py file describes to Distutils (1) how to package the distribution file and (2) 
how to install the distribution. Here is the contents of our sample setup.py: 


#!/usr/bin/env python 
from distutils.core import setup # [1] 


longudesicriperom — "Tests for anstalling and distributing Pyehou 
packages' 


setup(name = 'testpackages', # [2] 
version = ila", 
description = 'Tests for Python packages', 
maintainer = 'Dave Kuhlman', 
maintainer_email = 'dkuhlman@rexx.com', 
url = 'http://www.rexx.com/~dkuhlman', 
long_description = long_description, 
packages = ['testpackages'] # [3] 
) 


Explanation: 


1. We import the necessary component from Distutils. 

2. We describe the package and its developer/maintainer. 

3. We specify the directory that is to be installed as a package. When the user 
installs our distribution, this directory and all the modules in it will be installed as 
a package. 
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to create a distribution file, we run the following: 


python setup.py sdist --formats=gztar 


which will create a file testpackages—1.0a.tar.gz under the directory dist. 


Then, you can give this distribution file to a potential user, who can install it by doing the 
following: 

S tar xvzf testpackages—1.0a.tar.gz 

S$ cd testpackages-1.0a 

cS pyEnon Serup.oy louie 

S python setup.py install # as root 


2.9 End Matter 


2.9.1 Acknowledgements and Thanks 


Thanks to the implementors of Python for producing an exceptionally usable and 
enjoyable programming language. 

Thanks to Dave Beazley and others for SWIG and PLY. 

Thanks to Greg Ewing for Pyrex and Plex. 

Thanks to James Henstridge for PyGTK. 


2.9.2 See Also 


The main Python Web Site -- http://www.python.org for more information on 
Python. 

Python Documentation -- http://www.python.org/doc/ for lots of documentation 
on Python 

Dave's Web Site -- http://http://www.davekuhlman.org for more software and 
information on using Python for XML and the Web. 

The SWIG home page -- http://www.swig.org for more information on SWIG 
(Simplified Wrapper and Interface Generator). 

The Pyrex home page -- http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/ 
for more information on Pyrex. 

PLY (Python Lex-Yacc) home page -- http://www.dabeaz.com/ply/ for more 
information on PLY. 

The Plex home page -- http://www.cosc.canterbury.ac.nz/greg.ewing/python/Plex/ 
for more information on Plex. 

Distributing Python Modules -- http://docs.python.org/distutils/index.html for 
information on the Python Distribution Utilities (Distutils). 
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3 Part 3 -- Python Workbook 


3.1 Introduction 


This document takes a workbook and exercise-with-solutions approach to Python 
training. It is hoped that those who feel a need for less explanation and more practical 
exercises will find this useful. 


A few notes about the exercises: 


e I've tried to include solutions for most of the exercises. Hopefully, you will be 
able to copy and paste these solutions into your text editor, then extend and 
experiment with them. 

e [use two interactive Python interpreters (although they are the same Python 
underneath). When you see this prompt >>>, it's the standard Python interpreter. 
And, when you see this prompt In [1] :, it's [Python - 
http://ipython.scipy.org/moin/. 

The latest version of this document is at my Web site (URL above). 


If you have comments or suggestions, please send them my way. 


3.2 Lexical Structures 


3.2.1 Variables and names 


A name is any combination of letters, digits, and the underscore, but the first character 
must be a letter or an underscore. Names may be of any length. 


Case is significant. 
Exercises: 


1. Which of the following are valid names? 


l. total 

2. total_of_all_vegetables 
3. Bageememen 

4. _inner func 

5. lbigtitle 


6. bigtitlel 
2. Which or the following pairs are the same name: 
1. the_last_itemandthe_last_item 
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2. the_last_itemand The_Last_Item 
3. itemi and item} 
4. iteml and item] 

Solutions: 


1. Items 1, 2, 4, and 6 are valid. Item 3 is not a single name, but is three items 
separated by the minus operator. Item 5 is not valid because it begins with a digit. 
2. Python names are case-sensitive, which means: 
1. the_last_item and the_last_item are the same. 
2. the_last_itemand The_Last_Item are different -- The second name 
has an upper-case characters. 
itemi and item) are different. 

4. item1 and iteml are different -- This one may be difficult to see, 
depending on the font you are viewing. One name ends with the digit one; the 
other ends with the alpha character "el". And this example provides a good 
reason to use "1" and "1" judiciously in names. 

The following are keywords in Python and should not be used as variable names: 


na 


and 6lal micOml not while 
as elif global One with 
assert else aie pass yield 
break execu import joneabiane: 
class exec aja raise 
continue finally aL) 1S ic Ulacial 
Gleue ovis lambda ery 

Exercises: 


1. Which of the following are valid names in Python? 


1. _global 

2. global 

3. file 
Solutions: 


1. Do not use keywords for variable names: 

1. Valid 

2. Nota valid name. "global" is a keyword. 

3. Valid, however, "file" is the name of a built-in type, as you will learn later, so 
you are advised not to redefine it. Here are a few of the names of built-in 
types: "file", "int", "str", "float", "list", "dict", etc. See Built-in Types -- 
http://docs.python.org/lib/types.html for more built-in types.. 

The following are operators in Python and will separate names: 


+ - * oie / // % 
<< >> & as 
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< S = SS == t= <> 
and or is not sietal 
ALSO S () [] 5. (cloie)) 


But, note that the Python style guide suggests that you place blanks around binary 
operators. One exception to this rule is function arguments and parameters for functions: 
it is suggested that you not put blanks around the equal sign (=) used to specify keyword 
arguments and default parameters. 


Exercises: 


1. Which of the following are single names and which are names separated by 
operators? 
l. fruveseollection 
2. Cue collecrnen 
Solutions: 


1. Do not use a dash, or other operator, in the middle of a name: 
1. fruit_collection isa single name 
2. fruit-—collection is two names separated by a dash. 


3.2.2 Line structure 


In Python, normally we write one statement per line. In fact, Python assumes this. 
Therefore: 


e Statement separators are not normally needed. 
e But, if we want more than one statement on a line, we use a statement separator, 
specifically a semi-colon. 
e And, if we want to extend a statement to a second or third line and so on, we 
sometimes need to do a bit extra. 
Extending a Python statement to a subsequent line -- Follow these two rules: 


1. If there is an open context, nothing special need be done to extend a statement 
across multiple lines. An open context is an open parenthesis, an open square 
bracket, or an open curly bracket. 

2. Wecan always extend a statement on a following line by placing a back slash as 
the last character of the line. 

Exercises: 


1. Extend the following statement to a second line using parentheses: 


total_count = tree_count + vegetable_count + 
EGU Re OUT 


2. Extend the following statement to a second line using the backslash line 
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continuation character: 


total_count = tree_count + vegetable_count + 
ELUTE CouUnE 


Solutions: 


1. Parentheses create an open context that tells Python that a statement extends to 
the next line: 


E@eal Wecounte — (eEreceticount = 
vegetable_count + fruit_count) 


2. A backslash as the last character on line tells Python that the current statement 
extends to the next line: 


total_count = tree_count + \ 
vegetable_count + fruit_count 


For extending a line on a subsequent line, which is better, parentheses or a backslash? 
Here is a quote: 

"The preferred way of wrapping long lines is by using Python's implied 

line continuation inside parentheses, brackets and braces. If necessary, 

you can add an extra pair of parentheses around an expression, but 

sometimes using a backslash looks better." 


-- PEP 8: Style Guide for Python Code -- 
http://www.python.org/dev/peps/pep-0008/ 


3.2.3 Indentation and program structure 


Python uses indentation to indicate program structure. That is to say, in order to nest a 
block of code inside a compound statement, you indent that nested code. This is different 
from many programming languages which use some sort of begin and end markers, for 
example curly brackets. 


The standard coding practice for Python is to use four spaces per indentation level and to 
not use hard tabs. (See the Style Guide for Python Code.) Because of this, you will want 
to use a text editor that you can configure so that it will use four spaces for indentation. 
See here for a list of Python-friendly text editors: PythonEditors. 


Exercises: 


1. Given the following, nest the print statement inside the if statement: 


age ones ale 


[Owes OX 


2. Nest these two lines: 
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Tet, 
eM 


inside the following function definition statement: 


def show_sum(x, y): 


Solutions: 


1. Indentation indicates that one statement is nested inside another statement: 


ahin Se S2 (0)8 
jouealae, 9x 


2. Indentation indicates that a block of statements is nested inside another statement: 


def show_sum(x, y): 
Z=xt+y 
joieaine, 4 


3.3 Execution Model 


Here are a few rules: 


1. Python evaluates Python code from the top of a module down to the bottom of a 
module. 

2. Binding statements at top level create names (and bind values to those names) as 
Python evaluates code. Further more, a name is not created until it is bound to a 
value/object. 

3. A nested reference to a name (for example, inside a function definition or in the 
nested block of an if statement) is not used until that nested code is evaluated. 

Exercises: 


1. Will the following code produce an error? 


show_version() 
def show_version(): 
print ‘Version 1.0a' 


2. Will the following code produce an error? 


def test(): 
show_version() 


def show_version(): 
jonealigne, WWienesitoneys INO! 


test () 


3. Will the following code produce an error? Assume that show_config is not 
defined: 
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ee x SS Se 
show_config() 


Solutions: 


1. 


Answer: Yes, it generates an error. The name show_version would not be 
created and bound to a value until the def function definition statement binds a 
function object to it. That is done after the attempt to use (call) that object. 
Answer: No. The function test () does call the function show_version(), 
but since test () is not called until after show_version () is defined, that is 
OK. 

Answer: No. It's bad code, but in this case will not generate an error. Since x is 
less than 5, the body of the if statement is not evaluated. 

N.B. This example shows why it is important during testing that every line of 
code in your Python program be evaluated. Here is good Pythonic advice: "If it's 
not tested, it's broken." 


3.4 Built-in Data Types 


Each of the subsections in this section on built-in data types will have a similar structure: 


1. 
2 


3. 
4, 


A brief description of the data type and its uses. 

Representation and construction -- How to represent an instance of the data type. 
How to code a literal representation that creates and defines an instance. How to 
create an instance of the built-in type. 

Operators that are applicable to the data type. 

Methods implemented and supported by the data type. 


3.4.1 Numbers 


The numbers you will use most commonly are likely to be integers and floats. Python 
also has long integers and complex numbers. 


A few facts about numbers (in Python): 


Python will convert to using a long integer automatically when needed. You do 
not need to worry about exceeding the size of a (standard) integer. 

The size of the largest integer in your version of Python is in sys.maxint. To 
learn what it is, do: 


SSS alimeroudic, Sys 
22 Ones iale TShifs) guinebcilahe 
9223372036854775807 


The above show the maximum size of an integer on a 64-bit version of Python. 
You can convert from integer to float by using the float constructor. Example: 
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Soo x = 25 

>>> y = float (x) 
Se Pram ay 
25.50 


e Python does "mixed arithmetic". You can add, multiply, and divide integers and 
floats. When you do, Python "promotes" the result to a float. 
3.4.1.1 Literal representations of numbers 


An integer is constructed with a series of digits or the integer constructor (int (x) ). Be 
aware that a sequence of digits beginning with zero represents an octal value. Examples: 


>>> xl = 1234 

SS 2 line aL sii) 
>>> x3 = -25 

SS sell 

Lae 

SSS x2 

1234 

SS cS 

=25 


A float is constructed either with digits and a dot (example, 12.345) or with 
engineering/scientific notation or with the float constructor (float (x) ). Examples: 


>>> xl = 2.0e3 

>o> xl = 1.234 

SS, SAS! 

SS 8) tilkoate (aks 2 3) 

>>> x4 = 2.0e3 

2S> XS = 2.0e—-3 

Po> prime xi, Ka, KS, «4, XS 
ieZoa ot 234 ZOE 030s O02 


Exercises: 
Construct these numeric values: 


1. Integer zero 

2. Floating point zero 

3. Integer one hundred and one 

4. Floating point one thousand 

5. Floating point one thousand using scientific notation 

6. Create a positive integer, a negative integer, and zero. Assign them to variables 
7. Write several arithmetic expressions. Bind the values to variables. Use a variety 
of operators, e.g. +, —, /, *, etc. Use parentheses to control operator scope. 
Create several floats and assign them to variables. 

Write several arithmetic expressions containing your float variables. 


ce 
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10. Write several expressions using mixed arithmetic (integers and floats). Obtain a 
float as a result of division of one integer by another; do so by explicitly 
converting one integer to a float. 


Solutions: 

1. 0 

2. Omo, OB, or BO 

3. ional 

4. 1000.0 

5. le3orl.0e3 

6. Asigning integer values to variables: 
In [7]: valuel = 23 
In [8]: value2 = -14 
isa [S's weulues! = 0) 
en lO valuel 
Owe LO s 23 
Ia is weulvieZ 
Out[11]: -14 
Ia As welvies 
Omelieie C 

7. Assigning expression values to variables: 
valuel = 4 * (3 + 5) 
wellviee = Grellwel / 3.10) = 2 


8. Assigning floats to variables: 


valuel = 0.01 
value2 
value3 = 3e-4 


ll 

| 
Ww 
S 


9. Assigning expressions containing varialbes: 


value4 = valuel * (value2 —- value3) 
value4 = valuel + value2 + value3 value4 


10. Mixed arithmetic: 


x = 5 
y = 8 
Z — ilo (x) 9/4 


You can also construct integers and floats using the class. Calling a class (using 
parentheses after a class name, for example) produces an instance of the class. 


Exercises: 


1. Construct an integer from the string "123". 

2. Construct a float from the integer 123. 

3. Construct an integer from the float 12.345. 
Solutions: 
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1. Use the int data type to construct an integer instance from a string: 


abioue, (Mal si) 


2. Use the float data type to construct a float instance from an integer: 


ie AWoyele (i255) 


3. Use the int data type to construct an integer instance from a float: 


aba (L614 3) i? ae Le 


Notice that the result is truncated to the integer part. 


3.4.1.2 Operators for numbers 


You can use most of the familiar operators with numbers, for example: 


y = e = / // % 
KEG SS & | o ~ 
< > g= >— == — => 


Look here for an explanation of these operators when applied to numbers: Numeric 
Types -- int, float, long, complex -- http://docs.python.org/lib/typesnumeric.html. 


Some operators take precedence over others. The table in the Web page just referenced 
above also shows that order of priority. 


Here is a bit of that table: 


All numeric types (except complex) support the following operations, 

sorted by ascending priority (operations in the same box have the 

same 

priority; all numeric operations have a higher priority than 

comparison 

operations): 

Operation Result 

a ye sum of x and y 

De difference of x and y 

2 AY WROCUCE Oi =x satel y, 

a ay quotient of x and y 

e/a) (floored) quotient of x and y 

Bony: remainder of x / y 

=e x negated 

apo x unchanged 

abs (x) absolute value or magnitude of x 

int (x) x converted to integer 

long (x) x converted to long integer 

float (x) x converted to floating point 

complex(re,im) a complex number with real part re, imaginary part 
im) an demas: ol Zeno; 

c.conjugate() conjugate of the complex number c 
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Glingmeea(scymays) ilove’ jolie (Gx /7/ Wer e 
pow(x, y) x to the power y 
oe NE x to the power y 


Notice also that the same operator may perform a different function depending on the 
data type of the value to which it is applied. 


Exercises: 


1. Add the numbers 3, 4, and 5. 

2. Add 2 to the result of multiplying 3 by 4. 

3. Add 2 plus 3 and multiply the result by 4. 
Solutions: 


1. Arithmetic expressions are follow standard infix algebraic syntax: 


Syne gra) 


2. Use another infix expression: 


2 ate Bs 4 


Or: 


2+ (3 * 4) 


But, in this case the parentheses are not necessary because the * operator binds 
more tightly than the + operator. 
3. Use parentheses to control order of evaluation: 


(Zea Sy 


Note that the * operator has precedence over (binds tighter than) the + operator, 
so the parentheses are needed. 
Python does mixed arithemetic. When you apply an operation to an integer and a float, it 
promotes the result to the "higher" data type, a float. 


If you need to perform an operation on several integers, but want use a floating point 
operation, first convert one of the integers to a float using float (x), which effectively 
creates an instance of class float. 


Try the following at your Python interactive prompt: 


ii .@ + 2 

2. 2 / 3-- Notice that the result is truncated. 

3. float(2) / 3 -- Notice that the result is not truncated. 
Exercises: 


1. Given the following assignments: 


20 
50 


ss 
Ay 
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Divide x by y giving a float result. 
Solutions: 


1. Promote one of the integers to float before performing the division: 


| Za rel aten a) 0/52 


3.4.1.3 Methods on numbers 


Most of the methods implemented by the data types (classes) int and float are special 
methods that are called through the use of operators. Special methods often have names 
that begin and end with a double underscore. To see a list of the special names and a bit 
of an indication of when each is called, do any of the following at the Python interactive 
prompt: 


>>> help (int) 

>>> help (32) 

See jal) (ae ieee) 

See helpi(ie 2s) 

SS chia (i) 

SSS elie (1152) 
3.4.2 Lists 


Lists are a container data type that acts as a dynamic array. That is to say, a list is a 
sequence that can be indexed into and that can grow and shrink. 


A tuple is an index-able container, like a list, except that a tuple is immutable. 
A few characteristics of lists and tuples: 


e A list has a (current) length -- Get the length of a list with len (mylist). 

e A list has an order -- The items in a list are ordered, and you can think of that 
order as going from left to right. 

e A list is heterogeneous -- You can insert different types of objects into the same 
list. 

e Lists are mutable, but tuples are not. Thus, the following are true of lists, but not 
of tuples: 
o Youcan extended or add to a list. 
o Youcan shrink a list by deleting items from it. 
o Youcan insert items into the middle of a list or at the beginning of a list. You 

can add items to the end of a list. 

o Youcan change which item is at a given position in a list. 
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3.4.2.1 Literal representation of lists 


The literal representation of a list is square brackets containing zero or more items 
separated by commas. 


Examples: 


1. Try these at the Python interactive prompt: 


See lil 22, 33] 

Seo ilar blo, ica | 

2 (1007) Yapplle’, 2007 “banana, |] # The last comma 
acs 

>>> optional. 


2. A list can contain lists. In fact a list can contain any kind of object: 


See ide ap Sie #7 Wee G2 te ie 8 


3. Lists are heterogenous, that is, different kinds of objects can be in the same list. 
Here is a list that contains a number, a string, and another list: 


Exercises: 


SS 


[es Veione! 7 Hoe, seh] 


1. Create (define) the following tuples and lists using a literal: 


1. 


SIDAKWAWH 


1. 
A 


eo mM 


A tuple of integers 

A tuple of strings 

A list of integers 

A list of strings 

A list of tuples or tuple of lists 

A list of integers and strings and tuples 
A tuple containing exactly one item 
An empty tuple 


Do each of the following: 


Print the length of a list. 

Print each item in the list -- Iterate over the items in one of your lists. Print 
each item. 

Append an item to a list. 

Insert an item at the beginning of a list. Insert an item in the middle of a list. 
Add two lists together. Do so by using both the extend method and the plus 
(+) operator. What is the difference between extending a list and adding two 
lists? 

Retrieve the 2nd item from one of your tuples or lists. 

Retrieve the 2nd, 3rd, and 4th items (a slice) from one of your tuples or lists. 
Retrieve the last (right-most) item in one of your lists. 

Replace an item in a list with a new item. 
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10. Pop one item off the end of your list. 
11. Delete an item from a list. 
12. Do the following list manipulations: 

1. Write a function that takes two arguments, a list and an item, and that 

appends the item to the list. 

2. Create an empty list, 

Call your function several times to append items to the list. 
4. Then, print out each item in the list. 

Solutions: 


a 


1. We can define list literals at the Python or [Python interactive prompt: 
1. Create a tuple using commas, optionally with parentheses: 


Trae (il ad =" Cini 2. Say) 
iTisranens liz ll se ciel 
Oh | 2s (alle 22) 38) 


2. Quoted characters separated by commas create a tuple of strings: 


im [Sit az =“Craaat, “bbb",. Vece™) 
igh [4 |e) 2 
OUEA4I (aaa) bbb! 7. ‘ece') 


3. Items separated by commas inside square brackets create a list: 


im (Zo) 3 as = [L0C, 200, S00, | 
Ian (278 ais} 
GWE 27s (LOC, 200, S00] 


4. Strings separated by commas inside square brackets create a list of strings: 


In [5]¢ a3 = [“basil", “parsiley', ‘coriander" | 
tia (6 a3 

CuelG| = [beasti", “parsley”; Vcoritander* | 

ign ite 8 


5. A tuple or a list can contain tuples and lists: 


im ele ves al, 227 (S37 44) (S54 
in 9) 8 aS 
@wie Oe lil, 224), (8S. 44> (Ser) ] 


6. A list or tuple can contain items of different types: 


Tia PALO Sere — OS Oe ele Wickens C2 Gals 2102s) = 
(CYigtaal  alsil/)) 

iia [ie ais 
CUE ss Waa OL Valet e  Weleie ss AG 21025 
(Cigha aie )s| 


7. In order to create a tuple containing exactly one item, we must use a comma: 


iia (IS s a7 = (6,)) 
iigh || eo a7 
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Oey ee Cor) 


8. In order to create an empty tuple, use the tuple class/type to create an instance 
of a empty tuple: 


im [20 | a = tuple () 

igh [AZ ei 

QUE IZAZ EO) 

In [23]: type (a) 
QUEIZSIE <bype  euplet 


3.4.2.2 Operators on lists 


There are several operators that are applicable to lists. Here is how to find out about 
them: 


e Dodir([]) ordir(any_list_instance). Some of the items with 
special names (leading and training double underscores) will give you clues about 
operators implemented by the list type. 

e Dohelp([]) orhelp(list) at the Python interactive prompt. 

e Dohelp(any_list_instance.some_method), where some_method 
is one of the items listed using dir (any_list_instance). 

e See Sequence Types -- str, unicode, list, tuple, buffer, xrange -- 
http://docs.python.org/lib/typesseq. html 

Exercises: 


1. Concatenate (add) two lists together. 
2. Create a single list that contains the items in an initial list repeated 3 times. 
3. Compare two lists. 

Solutions: 


1. The plus operator, applied to two lists produces a new list that is a concatenation 
of two lists: 


Sep il, 22 ar leet 7 Visio] 


2. Multiplying a list by an integer n creates a new list that repeats the original list n 
times: 


Se aloe 4 Sie 1S 


3. The comparison operators can be used to compare lists: 


>>>. Aes 225) == s a 22) 
SSS [iil, 22 < ik, 333] 


3.4.2.3 Methods on lists 
Again, use dir () and help () to learn about the methods supported by lists. 
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1. Create two (small) lists. Extend the first list with the items in the second. 

2. Append several individual items to the end of a list. 

3. (a) Insert a item at the beginning of a list. (b) Insert an item somewhere in the 
middle of a list. 

4. Pop an item off the end of a list. 


Solutions: 


1. The extend method adds elements from another list, or other iterable: 


2. Use the 


Seo en = le ee se | 
So bp — So.) ion 

>>> a.extend(b) 

SSS a 


[de 225 Ss Bae San 6 


append method on a list to add/append an item to the end of a list: 


eee a = | alate 
>>> a.append('bb') 
>>> a.append (22) 
SSS él 


[vaal, i, Yolo”, 22] 


3. The insert method on a list enables us to insert items at a given position in a 


list: 


Ses 6 = [ill, 22, 23, aa, | 
>>> a.insert(0, 'aa') 
SSS a 


[rata pe le Se S84] 

Ses ely dlinsrevere (45 Vole} )) 

SS > el 

[Maat 1h bol, 22, 383i, 445] 


But, note that we use append to add items at the end of a list. 
4. The pop method on a list returns the "right-most" item from a list and removes 
that item from the list: 


SS > ae= lilly 227 B37 3447 
SS 

>>> b = a.pop() 
SS> a 

[il 22, 33] 
SSS I) 

44 

>>> b = a.pop() 
SS Sal 

[ai 22] 

SS Is) 

Sis) 


Page 179 


A Python Book 


Note that the append and pop methods taken together can be used to implement 
a stack, that is a LIFO (ast in first out) data structure. 


3.4.2.4 List comprehensions 


A list comprehension is a convenient way to produce a list from an iterable (a sequence 
or other object that can be iterated over). 


In its simplest form, a list comprehension resembles the header line of a for statement 
inside square brackets. However, in a list comprehension, the for statement header is 
prefixed with an expression and surrounded by square brackets. Here is a template: 


[expr(x) for x in iterable] 


where: 


e expr (x) is an expression, usually, but not always, containing x. 

e iterable is some iterable. An iterable may be a sequence (for example, a list, a 
string, a tuple) or an unordered collection or an iterator (something over which we 
can iterate or apply a for statement to). 

Here is an example: 


el | sy | 
SS lo Se ieioie Se akin i] 
SSS Is 


[227 447 6G, Sc] 


Exercises: 


1. Given the following list of strings: 


mames = [Vallaice’, “bertrand’, “chanilene” | 


produce the following lists: (1) a list of all upper case names; (2) a list of 
capitalized (first letter upper case); 
2. Given the following function which calculates the factorial of a number: 


der E(n) + 
if n <=1: 
return n 
else: 
ene Wai says ie (ial 1) 


and the following list of numbers: 


numbers = [|2, 3, 4, 5] 


create a list of the factorials of each of the numbers in the list. 
Solutions: 


1. For our expression in a list comprehension, use the upper and capitalize 
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methods: 


SS eames — a) vallkneet  Voerincand 7) Veihaaileme || 
>>> [name.upper() for name in names] 
['ALTICE', "BERTRAND', 'CHARLENE' ] 

>>> [name.capitalize() for name in names] 
["Allace", “Bertrand , "“Charilene” | 


2. The expression in our list comprehension calls the factorial function: 


dem f(m) + 
dof “<a 
return n 


else: 
TeeVeieim int “sie (Ga = 1) 
def test(): 
numbers = |2, 3, 4, 5] 
ieleneoue wells) =" [ie (ial) “iglesia alia) sanbhmalorese's)) 


pein tactormialss\ wactormaails 


aie name == '  main__': 


A list comprehension can also contain an if clause. Here is a template: 


[expr(x) for x in iterable if pred(x) ] 


where: 


e@ pred(x) is an expression that evaluates to a true/false value. Values that count 
as false are numeric zero, False, None, and any empty collection. All other 
values count as true. 


Only values for which the if clause evaluates to true are included in creating the resulting 
list. 


Examples: 


Seo ya = (225. 337-744) 

SS = se 4 92) orm ina ee 2 = — 0] 
SS = |9 

GG, they? | 


Exercises: 


1. Given two lists, generate a list of all the strings in the first list that are not in the 
second list. Here are two sample lists: 


namesl 
names2 


| Velllaveet — Uioeicigicenael! -  eloeuellaines 3 Vicleiaedl || 
['bertrand', 'charlene'] 


Solutions: 


1. The if clause of our list comprehension checks for containment in the list names2: 
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def test(): 

namesl = ['alice', 'bertrand', 'charlene', 
Vdemucie | 

names2 = ['bertrand', 'charlene'] 

names3 = [name for name in namesl if name not in 
names2] 


print 'names3:', names3 


aie name == '  main__': 
test () 


When run, this script prints out the following: 


Mamessr i vellaicel damaiein | 


3.4.3 Strings 


A string is an ordered sequence of characters. Here are a few characteristics of strings: 


e A string has a length. Get the length with the len () built-in function. 
e A string is indexable. Get a single character at a position in a string with the 
square bracket operator, forexample mystring[5]. 
e Youcan retrieve a slice (sub-string) of a string with a slice operation, for example 
MN SIE 12 LING [5 3S IF 
Create strings with single quotes or double quotes. You can put single quotes inside 
double quotes and you can put double quotes inside single quotes. You can also escape 
characters with a backslash. 


Exercises: 


1. Create a string containing a single quote. 

2. Create a string containing a double quote. 

3. Create a string containing both a single quote a double quote. 
Solutions: 


1. Create a string with double quotes to include single quotes inside the string: 


Se Scilly = MVicleene als ajenemya Ss joi 


2. Create a string enclosed with single quotes in order to include double quotes 
inside the string: 


>>> strl = 'say "goodbye", bullwinkle' 
3. Take your choice. Escape either the single quotes or the double quotes with a 
backslash: 
Soo Sil — say ehellloW Neon gieiriay wis mom 
>>> stu2 — “say \“helilo\" too jerry s mom" 
See Siciell 
lsany Winellie eo aeicieye\ Vs. inlolaa! 
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Se Scie! 
Vsay une lslow Teougierin Vase mon 


Triple quotes enable you to create a string that spans multiple lines. Use three single 
quotes or three double quotes to create a single quoted string. 


Examples: 


1. Create a triple quoted string that contains single and double quotes. 
Solutions: 


1. Use triple single quotes or triple double quotes to create multi-line strings: 


Stringl = '''This string extends 
across several lines. And, so it has 
end-of-line characters in it. 


1 ae ae § 


Strimg2 Si 
This string begins and ends with an end-of-line 
character. It can have both 'single' 


quotes and "double" quotes in it. 


WW 


def test(): 
joneniahe, Sie i@aliave jl 
(ue Me, Sie ican 
alae name == ' | main__': 
test () 


3.4.3.1 Characters 


Python does not have a distinct character type. In Python, a character is a string of length 
1. You can use the ord() and chr () built-in functions to convert from character to 
integer and back. 


Exercises: 


1. Create a character "a". 
2. Create a character, then obtain its integer representation. 
Solutions: 


1. The character "a" is a plain string of length 1: 


SoS x = Tait 


2. The integer equivalent of the letter "A": 


Sb ee a ULM 
SSS wc! (F<) 
65 


Page 183 


A Python Book 


3.4.3.2 Operators on strings 

You can concatenate strings with the "+" operator. 

You can create multiple concatenated copies of a string with the "*" operator. 
And, augmented assignment (+= and *=) also work. 


Examples: 


See Verne! cr! eharel 4 on Nolo)! 
Vierhie ~linvel ilove ¥ 
SS A) 
EEE HE HE HE HE EE HE EH HE HERE EE REE EE EERE EEE | 
a 

>>> sl = 'flower' 

See ell eS Ns! 

SS Sil 

"flowers' 


Exercises: 


1. Given these strings: 


SS os l=] Vabed! 
>>> s2 = 'efgh' 


create a new string composed of the first string followed by (concatenated with) 

the second. 

2. Create a single string containing 5 copies of the string ‘abc’. 
Use the multiplication operator to create a "line" of 50 dashes. 

4. Here are the components of a path to a file on the file system: "home", 
"myusername", "Workdir", "notes.txt". Concatenate these together separating 
them with the path separator to form a complete path to that file. (Note that if you 
use the backslash to separate components of the path, you will need to use a 
double backslash, because the backslash is the escape character in strings. 

Solutions: 


al 


1. The plus (+) operator applied to a string can be used to concatenate strings: 


Se SS = Sl a SZ 
SS 2 
vabedeign * 


2. The multiplication operator (*) applied to a string creates a new string that 
concatenates a string with itself some number of times: 


Se> Bll = Valse! = 5 
>>> sl 
"abcabcabcabcabc' 


3. The multiplication operator (*) applied to a string can be used to create a 
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"horizontal divider line": 


SSeS = 8S) = 50 
SSP jolene sil 


4. The sep member of the os module gives us a platform independent way to 
construct paths: 


Se alileyencie. ys 

>>> 

>>> a = ["home", "myusername", "Workdir", "notes.txt"] 
SS> selell = alll a OSssiejo a ali) a OSaseio a a2 a 
os.sep + a[3] 

2 > > paeh 

"home/myusername/Workdir/notes.txt' 


And, a more concise solution: 


>>> import os 

>>> a = ["home", "myusername", "Workdir", "notes.txt"] 
>>> os.sep.join(a) 

"home/myusername/Workdir/notes.txt' 


Notes: 

o Note that importing the os module and then using os. sep from that module 
gives us a platform independent solution. 

o If you do decide to code the path separator character explicitly and if you are 
on MS Windows where the path separator is the backslash, then you will need 
to use a double backslash, because that character is the escape character. 


3.4.3.3 Methods on strings 


String support a variety of operations. You can obtain a list of these methods by using the 
dir () built-in function on any string: 


SS Clikie (0) 

(ee adda. | class vibes Gomceiuas 9p !_ clellenciee ea Emel (ove! a 
"eq." , tooage.',, "9getattrabuve. ", *_ qétitemo', 

VEG CENeWacds a meager olinicei |. “Sigtee. |) 'Seshcsiteet yeni, 
: i = ee len es Ie wp) Siecle samen 7. Vane ep 
renew. | reduce); ‘) rednce ex", * repr i TEMMENC fe 
‘srmuil "7, “Ss asetattir =!) “ssstre , ‘capitalize’. center , 


"count', 'decode', ‘encode', 'endswith', 'expandtabs', 'find', 
‘index', ‘isalnum', 'isalpha', 'isdigit', 'islower', ‘isspace', 
Diksieatieie 5 Uaksibcjorerel  Saioslin! - Vile) kone 2 Y Ieierestiol 
Jiovseicanieayoial 7 Vaesjodleicey 5° YiciealiovelY 4) Usealioveler 5) Vie ibis! 5) /iaevchaicalic atonal, 7 
Jsejollanicve iescieabjoy Vs ovlliskie 7 sjoulslicdbakiaveysry pa Visneehcicieiiariola Vis cke@akioy 
“Swapcase', “title, “Eransilatre', “uppern’, “ztadd'| 


And, you can get help on any specific method by using the he 1p () built-in function. 
Here is an example: 
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Se inieiljo (VY Sicieiia)) 
Helpmon built ain —funeCrTon Stel: 


Sesackon Gas oy) 
Se serrpehads|)i = seal mo tor unieode 


Return a copy of the string S with leading and trailing 

whitespace removed. 

If chars is given and not None, remove characters in chars 
instead. 

If chars is unicode, S will be converted to unicode before 
stripping 


Exercises: 


1. Strip all the whitespace characters off the right end of a string. 

2. Center a short string within a longer string, that is, pad a short string with blank 
characters on both right and left to center it. 

3. Convert a string to all upper case. 

Split a string into a list of "words". 
5. (a) Join the strings in a list of strings to form a single string. (b) Ditto, but put a 

newline character between each original string. 
Solutions: 


- 


1. The rstrip() method strips whitespace off the right side of a string: 


SSS sil = 'sone icezc Nn 
SS SL 

"some text Nae! 

22 B= Sil aie Siero) (() 

SS 2 

"some text' 


2. The center (n) method centers a string within a padded string of width n: 


>>> sl = 'Dave' 

>>> s2 = sl.center (20) 
SS se 

y Dave 4 


3. The upper () method produces a new string that converts all alpha characters in 
the original to upper case: 


>>> sl = 'Banana' 
SSS sil 

"Banana' 

>>> s2 = sl.upper() 
SSS S72 

"BANANA' 


4. The split (sep) method produces a list of strings that are separated by sep in 
the original string. If sep is omitted, whitespace is treated as the separator: 
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>>> sl = """how does it feel 
to be on your own 
no directions known 
like a rolling stone 
aa WoW 
Se yioneclsy = sil, Sjeulaine (() 
>>> words 
unowi. “does, Saint; “reel Miro"; Uber 7. one, eyoure, 
VOW 7. NON; 
Vdimmcctaions: |, “enown'; ‘lake, Vay, "sodium, “stone | 


Note that the split () function in the re (regular expression) module is useful 
when the separator is more complex than whitespace or a single character. 

5. The join() method concatenates strings from a list of strings to form a single 
string: 


>>> lines = [] 
>>> lines.append 
>>> lines.append 
>>> lines.append 
>>> lines.append 
>>> lines 
['how does it feel', 'to be on your own', ‘no 
directions known', 
‘like a rolling stone'] 


"how does it feel") 
"to be on your own') 
"no directions known') 
‘like a rolling stone') 


( 
( 
( 
( 


2 >= si Ve gouni(diines) 
2>> s2— * *. Jorn (lines) 
>>> s3 = '\n'.Jjoin(lines) 
>>>) cil 


"how does it feelto be on your ownno directions 
knownlike a rolling stone’ 

SS SZ 
"how does it feel to be on your own no directions known 
like a rolling stone’ 

SSS GS) 
"how does it feel\nto be on your own\nno directions 
known\nlike a rolling stone' 

SSS jellies SS 

how does it feel 

EQbe) On) ours own 

no directions known 

like a rolling stone 


3.4.3.4 Raw strings 


Raw strings give us a convenient way to include the backslash character in a string 
without escaping (with an additional backslash). Raw strings look like plain literal 
strings, but are prefixed with an "r" or "R". See String literals 
http://docs.python.org/reference/lexical_analysis.html#string-literals 


Excercises: 
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1. Create a string that contains a backslash character using both plain literal string 
and a raw string. 
Solutions: 


1. We use an "r" prefix to define a raw string: 


[> Prine tabe Weeder! 
abc \ def 
[> preints clalbc det! 
abc \ def 


3.4.3.5 Unicode strings 


Unicode strings give us a consistent way to process character data from a variety of 
character encodings. 


Excercises: 


1. Create several unicode strings. Use both the unicode prefix character ("u") and the 
unicode type (unicode (some_string) ). 


2. Convert a string (possibly from another non-ascii encoding) to unicode. 
3. Convert a unicode string to another encoding, for example, utf-8. 
4. Test a string to determine if it is unicode. 
5. Create a string that contains a unicode character, that is, a character outside the 
ascii character set. 
Solutions: 


1. We can represent unicode string with either the "u" prefix or with a call to the 
unicode type: 


def exercisel(): 
a = u'abcd' 
jones: 
b = unicode('efgh') 
jonestiahe: 16) 


2. We convert a string from another character encoding into unicode with the 
decode () string method: 


import sys 


def exercise2(): 


a = 'abcd'.decode('utf£-8') 

Pein a 

b = 'abcd'.decode (sys.getdefaultencoding() ) 
joneslinie. 16) 


3. Wecan convert a unicode string to another character encoding with the 
encode () string method: 


import sys 
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def exercise3(): 
ei = ul! eleree!! 
print a.encode('utf-8"') 
print a.encode(sys.getdefaultencoding() ) 


4. Here are two ways to check the type of a string: 


import types 


def exercise4(): 
2 = wl! elorere!" 
print type(a) is types.UnicodeType 
print type(a) is type(u'') 


5. Wecan encode unicode characters in a string in several ways, for example, (1) by 
defining a utf-8 string and converting it to unicode or (2) defining a string with an 
embedded unicode character or (3) concatenating a unicode characher into a 
string: 


def exercise5(): 
Weis) Sie icine) = Iie IGese al eal Noth 7! 
unicode_string = utf8_string.decode('utf-8"') 
print unicode_string.encode ('utf-8"') 
print len(utf8_string) 
print len(unicode_string) 
unicode_string = u'aa\u0107bb' 
print unicode_string.encode ('utf-8') 
Uunicodelstring — “aa unaehr (263) 4 uobY 
DLink Unicode string. cncode( uti—s") 


Guidance for use of encodings and unicode: 


1. Convert/decode from an external encoding to unicode early: 


my_source_string.decode (encoding) 


2. Do your work (Python processing) in unicode. 
3. Convert/encode to an external encoding late (for example, just before saving to an 
external file): 


my_unicode_string.encode (encoding) 


For more information, see: 


e Unicode In Python, Completely Demystified -- http://farmdev.com/talks/unicode/ 

Unicode How-to -- http://www.amk.ca/python/howto/unicode. 

e PEP 100: Python Unicode Integration -- 
http://www.python.org/dev/peps/pep-0100/ 

e 4.8 codecs -- Codec registry and base classes -- 
http://docs.python.org/lib/module-codecs.html 

e 4.8.2 Encodings and Unicode -- 
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http://docs.python.org/lib/encodings-overview.html 
e 4.8.3 Standard Encodings -- http://docs.python.org/lib/standard-encodings.html 
e Converting Unicode Strings to 8-bit Strings -- 
http://effbot.org/zone/unicode-convert.htm 


3.4.4 Dictionaries 
A dictionary is an un-ordered collection of key-value pairs. 
A dictionary has a length, specifically the number of key-value pairs. 


A dictionary provides fast look up by key. 


The keys must be immutable object types. 


3.4.4.1 Literal representation of dictionaries 


Curley brackets are used to represent a dictionary. Each pair in the dictionary is 
represented by a key and value separated by a colon. Multiple pairs are separated by 
comas. For example, here is an empty dictionary and several dictionaries containing 
key/value pairs: 


iia [4s cil = 4) 
Iie (LS) Sle atcha Se issn Movesabeleney al al | 
In Ca ae eae RED 2 GREEN oi VB In UE a. 
alitare ier ias elit 
Ouest 
Ina (Shs el 
Ole s a Moers sda Mykclela! ss} 2 5) 
iiah (PSNR sels) 
Out [9]: {l: 'RED', 2: 'GREEN', 3: 'BLUE'} 
Notes: 


e Acomma after the last pair is optional. See the RED-GREEN-BLUE example 
above. 

e Strings and integers work as keys, since they are immutable. You might also want 
to think about the use of tuples of integers as keys in a dictionary used to 
represent a sparse array. 

Exercises: 


1. Define a dictionary that has the following key-value pairs: 
2. Define a dictionary to represent the "enum" days of the week: Sunday, Monday, 
Tuesday, ... 
Solutions: 


1. A dictionary whose keys and values are strings can be used to represent this table: 


Page 190 


A Python Book 


vegetables = { 
Wne(ejolkeume ! 8 Views ellis < 
TROMaAsOu MRE 
"Parsley': 'Green', 
"Lemon': 'Yellow', 
"Pepper': 'Green', 
} 


Note that the open curly bracket enables us to continue this statement across 


multiple lines without using a backslash. 


2. We might use strings for the names of the days of the week as keys: 


DAYS = { 

"Sunday': i; 
"Monday': tar, 
'Tuesday': 3, 
"Wednesday': 4, 
PAL musical. =a o!, 
EN aalGlagyaua 6, 
UiSvateibliatelelyi vB | 7) 
ii 


3.4.4.2 Operators on dictionaries 


Dictionaries support the following "operators": 


e Length -- len (d) returns the number of pairs in a dictionary. 


e Indexing -- You can both set and get the value associated with a key by using the 


indexing operator [ ]. Examples: 


Wa LAs Cs 2 

Obie || LAI) Ss GABE 

Tiny Sa eels) LON) Sawer! 
abi | ibe) a ols) 10) 

Out [14] "WHITE' 


e Test for key -- The in operator tests for the existence of a key in a dictionary. 


‘cedar': 


Example: 
In [6]: trees = {'poplar': 'deciduous', 
"evergreen ' } 
In [7]: if 'cedar' in trees: 
eit print 'The cedar is 
(trees['cedar'], ) 
The cedar is evergreen 
Exercises: 


1. Create an empty dictionary, then use the indexing operator [ 


following name-value pairs: 


] to in sert the 


SEISYGUM | at WZ Os OOM 
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UNO re rare (OU 
MOO on 


"green" pa. 
Willie! = 


2. Print out the number of items in your dictionary. 
Solutions: 


1. Wecan use "[ ]" to set the value of a key in a dictionary: 


def test (): 
colons = 4} 
Coltows | red) = MASS OOr 
colors | “green” = "0225520" 
colons | olwe" = "00 3250)" 
print 'The value of red is "%s"' % 
(eollonss || Vieeel! |, )) 
print 'The colors dictionary contains %d items.' 
(len(colors), ) 
test () 
When we run this, we see: 
The value of red is "2552020" 
The colors dictionary contains 3 items. 


2. The len () built-in function gives us the number of items in a dictionary. See the 


previous solution for an example of this. 


3.4.4.3 Methods on dictionaries 


Here is a table that describes the methods applicable to dictionarys: 


Operation Result 
len(a) the number of items in a 
a[k] the item of a with key k 
a[k] =v set a[k] to v 
del a[k] remove a[k] from a 
a.clear() remove all items from a 
a.copy() a (shallow) copy of a 
kina True if a has a key k, else False 
knotina equivalent to not k ina 


a.has_key(k) 


equivalent to k in a, use that form in new code 


a.items() 


a copy of a's list of (key, value) pair 
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Operation 


Result 


a.keys() 


a copy of a's list of keys 


a.update([b]) 


updates a with key/value pairs from b, overwriting existing 
keys, returns None 


a.fromkeys(seq[, value]) 


creates a new dictionary with keys from seq and values set to 
value 


a.values() 


a copy of a's list of values 


a.get(k[, x]) 


a[k] if k in a, else x) 


a.setdefault(k[, x]) 


a[k] if k in a, else x (also setting it) 


a.pop(kI, x]) 


a[k] if k in a, else x (and remove k) (8) 


a.popitem() 


remove and return an arbitrary (key, value) pair 


a.iteritems() 


return an iterator over (key, value) pairs 


a.iterkeys() 


return an iterator over the mapping's keys 


a.itervalues() 


return an iterator over the mapping's values 


You can also find this table at the standard documentation Web site in the "Python 
Library Reference": Mapping Types -- dict http://docs.python.org/lib/typesmapping.html 


Exercises: 


1. Print the keys and values in the above "vegetable" dictionary. 
2. Print the keys and values in the above "vegetable" dictionary with the keys in 


alphabetical order. 


3. Test for the occurance of a key in a dictionary. 


Solutions: 


1. Wecan use the d. items () method to retrieve a list of tuples containing 
key-value pairs, then use unpacking to capture the key and value: 


Vegetables = { 
Uibofejolkcume Yo We tleoilis: Y 
'Tomato': 'Red', 
WPeueslley' 3s ‘Giesei’ , 
"Lemon': 'Yellow', 
"Pepper': 'Green', 


Page 193 


A Python Book 


Clair ieee ()) 2 
for key, value in Vegetables.items(): 
print 'key:', key, ' value:', value 


test () 


2. We retrieve a list of keys with the keys () method, the sort it with the list 
sort () method: 


Vegetables = { 
Minefojolkcume ! 5 Ae lcjoulie ! 7 
“LOmMaico. = VRE, 
"Parsley': 'Green', 
"Lemon': 'Yellow', 
"Pepper': 'Green', 

} 
clei eesie ()) 3 


keys = Vegetables.keys() 
keys.sort () 
for key in keys: 
print 'key:', key, ' value:', Vegetables [key] 


test () 


3. To test for the existence of a key in a dictionary, we can use either the in 
operator (preferred) or the d. has_key () method (old style): 


Vegetables = { 
Wae(sjoleime! 8 Vetueellis\ 
Wonasherer es MV Igerell 
"Parsley': 'Green', 
"Lemon': 'Yellow', 
"Pepper': 'Green', 

} 
Giese eee ()) 3 


if 'Eggplant' in Vegetables: 
print 'we have %s egplants' % 
Wee Sieelolless ||“ iseiejolbeuate © || 
if 'Banana' not in Vegetables: 
print 'yes we have no bananas’ 
if Vegetables.has_key('Parsley'): 
print 'we have leafy, %s parsley' % 
Vegetables['Parsley'] 


test () 


Which will print out: 


we have Purple egplants 
yes we have no bananas 
we have leafy, Green parsley 
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3.4.5 Files 


A Python file object represents a file on a file system. 


A file object open for reading a text file is iterable. When we iterate over it, it produces 
the lines in the file. 


A file may be opened in these modes: 


e 'r' -- read mode. The file must exist. 
e 'w' -- write mode. The file is created; an existing file is overwritten. 
e ‘a’ -- append mode. An existing file is opened for writing (at the end of the file). A 
file is created if it does not exist. 
The open () built-in function is used to create a file object. For example, the following 
code (1) opens a file for writing, then (2) for reading, then (3) for appending, and finally 
(4) for reading again: 


def test (infilename): 
# 1. Open the file in write mode, which creates the file. 


outfile = open(infilename, 'w') 
outfile.write('line 1\n') 
outfile.write('line 2\n') 
Olio swabs aeakiere (! iLatioves 3) Nim \4)) 

( 


outfile.close() 

# 2. Open the file for reading. 
infile = open(infilename, 'r') 
for line in infile: 

peint “ames "7. lames rsier a) () 
infile.close() 
# 3. Open the file in append mode, and add a line to the end of 
# the file. 

outfile = open(infilename, 'a') 

outfile.write('line 4\n') 

CurEtile close) 


jonaulignes fl Sec Ai0) 

# 4. Open the file in read mode once more. 
infile = open(infilename, 'r') 

Scene IESLinvey Ghia) sinicaikes 


print "“hine:', line.rstrip() 
infile.close() 


Gest Emp Sex," ) 


Exercises: 


1. Open a text file for reading, then read the entire file as a single string, and then 
split the content on newline characters. 

2. Open a text file for reading, then read the entire file as a list of strings, where each 
string is one line in the file. 

3. Open a text file for reading, then iterate of each line in the file and print it out. 
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1. Use the open () built-in function to open the file and create a file object. Use the 
read () method on the file object to read the entire file. Use the split () or 


splitlines() methods to split the file into lines: 
>>> infile = open('tmp.txt', 'r') 
>>> content = infile.read() 
>>> infile.close() 
>>> lines = content.splitlines() 
See jenealiate Ibakmies 
[eyeliner bilemey crea Pate] 


2. The £.readlines () method returns a list of lines in a file: 


>>> 
ee 
>>> 
Doe 


PY lasees: a ia 


inkale — open (empresa! Ue") 
lines = infile.readlines () 
infile.close() 

print lines 

Ualaisave 2 Vine 


Visine: 3 \in!] 


3. Since a file object (open for reading) is itself an iterator, we can iterate over it ina 


for statement: 


def 


def 


ase 


Test iteration over a text file. 
Usage: 


python test.py in_file_name 


import sys 


test (infilename) : 

infile = open(infilename, 
saya, Maine) aliny Jalelseat ley? 

Strip off the new-line character and any 


vere) 


whitespace on 


ene mieaoiites, 

line = line.rstrip() 

Print only non-blank lines. 
ie dlames 

print line 

infile.close() 


main(): 

args = sys.argv[1:] 

if len(args) != 1: 
joueaache Cle 
sys.exit (1) 

infilename = args[0] 

test (infilename) 

name == '  main__': 

main() 
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Notes: 

co The last two lines of this solution check the __name___ attribute of the 
module itself so that the module will run as a script but will not run when the 
module is imported by another module. 

o The __doc__ attribute of the module gives us the module's doc-string, which 
is the string defined at the top of the module. 

Oo sys.argv gives us the command line. And, sys.argv[1:] chops off the 
program name, leaving us with the comman line arguments. 


3.4.6 A few miscellaneous data types 


3.4.6.1 None 


None is a singleton. There is only one instance of None. Use this value to indicate the 
absence of any other "real" value. 


Test for None with the identity operator is. 
Exercises: 


1. Create a list, some of whose elements are None. Then write a for loop that 
counts the number of occurances of None in the list. 
Solutions: 


1. The identity operators is and is not can be used to test for None: 


So> a = [I1l, None, ‘abe’; Nome, {7} ] 
SSS el 

(ak None, abe None, 5] 

>>> count = 0 


>>> ore ase em! em vals 
if item is None: 
count += 1 
>> 
2 oiealiane  exeubhghe 
Z 


3.4.6.2 The booleans True and False 


Python has the two boolean values True and False. Many comparison operators return 
True and False. 


Examples: 


1. What value is returned by 3 > 2? 
Answer: The boolean value True. 
2. Given these variable definitions: 
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x= 3 
y=4 
Zo H5 


What does the following print out: 


jonealinue yy S> e< elin\el aa) yy 


Answer -- Prints out "True" 


3.5 Statements 


3.5.1 Assignment statement 
The assignment statement uses the assignment operator =. 


The assignment statement is a binding statement: it binds a value to a name within a 
namespace. 


Exercises: 


1. Bind the value "eggplant" to the variable vegetable. 
Solutions: 


1. The = operator is an assignment statement that binds a value to a variable: 


>>> vegetable = "eggplant" 
There is also augmented assignment using the operators +=, -=, *=, /=, etc. 
Exercises: 


1. Use augmented assignment to increment the value of an integer. 

2. Use augmented assignment to append characters to the end of a string. 

3. Use augmented assignment to append the items in one list to another. 

4. Use augmented assignment to decrement a variable containing an integer by 1. 
Solutions: 


1. The += operator increments the value of an integer: 


o> count = 0 
[> Count. — 
2>> Count 


Soe count a 1 
ee Count 


2. The += operator appends characters to the end of a string: 


>>> buffer = 'abcde' 
2 1 oibeinisne ap acreel 
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>>> buffer 
Vlei ie) 


3. The += operator appends items in one list to another: 


In [20 ae (PlG, 22. 33)] 

ay PS oy eal Sy] 

eae ny aes a) 

Ia (P28 a 

Owie2s [ity 227.. 33, 44, 55] 


1. The -= operator decrements the value of an integer: 


2o> Count = 5 
2 SCOuUMe 


>> count —— Al 
Ze WOuLae 


You can also assign a value to (1) an element of a list, (2) an item in a dictionary, (3) an 


attribute of an object, etc. 


Exercises: 


1. Create a list of three items, then assign a new value to the 2nd element in the list. 
2. Create a dictionary, then assign values to the keys "vegetable" and "fruit" in that 


dictionary. 
3. Use the following code to create an instance of a class: 


class A(object): 
pass 
aS IA) 


Then assign values to an attribue named category in that instance. 


Solutions: 


1. Assignment with the indexing operator [] assigns a value to an element in a list: 


>>> trees = ['pine', 'oak', ‘'elm'] 
>>> trees 

[onic ee oale 7) eam 

>>> trees[1] = 'cedar' 

>>> trees 

['pine', 'cedar', ‘'elm'] 


2. Assignment with the indexing operator [] assigns a value to an item (a key-value 


pair) in a dictionary: 


>>> foods = {} 
SSS iTOOCs 


>>> foods['vegetable'] = 'green beans' 
SSS foods |! faut! | = mMectalciine® 
SSS woos 
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(vegetable oreen beans", 9) haut Ye Mec areame nt 
g g 


3. Assignment along with the dereferencing operator . (dot) enables us to assign a 
value to an attribute of an object: 


>>> class A(object): 
pass 

>>> a = A() 

>>> a.category = 25 

o> din Gee 

{MiGabegouy. + 251} 

Be INS eS Ci Oaiy 

29 


3.5.2 print statement 


Warning: Be aware that the print statement will go away in Python version 3.0. It will 
be replaced by the built-in print () function. 


The print statement sends output to standard output. It provides a somewhat more 
convenient way of producing output than using sys.stdout.write(). 


The print statement takes a series of zero or more objects separated by commas. Zero 
objects produces a blank line. 


The print statement normally adds a newline at the end of its output. To eliminate that, 
add a comma at the end. 


Exercises: 


1. Print a single string. 

2. Print three strings using a single print statement. 

3. Given a variable name containing a string, print out the string My name is 
"xxxx"., where xxxx is replace by the value of name. Use the string formatting 
operator. 

Solutions: 


1. We can print a literal string: 


SSS Oiesime VniSlile, elmer | 
Hello, there 


2. We can print literals and the value of variables: 


>>> description = 'cute' 
SSS pied, VI eit al, clesieajeycaloin, )ealels 
I am a cute kid. 


3. The string formatting operator gives more control over formatting output: 


>>> name = 'Alice' 
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>>> print 'My name is "%Ss".' % (name, ) 
My name is "Alice". 


3.5.3 if: statement exercises 


The if statement is a compound statement that enables us to conditionally execute 
blocks of code. 


The if statement also has optional elif: and else: clauses. 


The condition in an if: or elif: clause can be any Python expression, in other words, 
something that returns a value (even if that value is None). 


In the condition in an if: or elif: clause, the following values count as "false": 


alee 

None 

Numeric zero 

An empty collection, for example an empty list or dictionary 
e Anempty string (a string of length zero) 

All other values count as true. 


Exercises: 


1. Given the following list: 


>>> bananas = ['bananal', 'banana2', 'banana3"', ] 


Print one message if it is an empty list and another messge if it is not. 
2. Here is one way of defining a Python equivalent of an "enum": 


N©_COLOR, RED, GREEN, BLUE — range (4) 


Write an if: statement which implements the effect of a "switch" statement in 
Python. Print out a unique message for each color. 
Solutions: 


1. We can test for an empty or non-empty list: 


>>> bananas = ['bananal', 'banana2', 'banana3"', ] 
>e> Ar not bananas: 
print 'yes, we have no bananas’ 
else: 
print 'yes, we have bananas’ 


yes, we have bananas 


2. Wecan simulate a "switch" statement using if:elif: 


NO_COLOR, RED, GREEN, BLUE = range (4) 


def test (color): 
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if color == RED: 

joneabigne,. Wale rey eharexc il 

elif color == GREEN: 

Deine Vile ts green.” 

elif color == BLUE: 
Deine Vie ts: bles 


def main(): 
color = BLUE 
Pest (color) 


aL ie name == '  main__': 
main () 


Which, when run prints out the following: 


It's blue. 


3.5.4 for: statement exercises 


The for: statement is the Python way to iterate over and process the elements of a 
collection or other iterable. 


The basic form of the for: statement is the following: 


@stgy | OX ata Vrs 
statement 
fe) 
fe) 
fe) 


where: 


e X is something that can be assigned to. It is something to which Python can bind a 


value. 
e Yissome collection or other iterable. 
Exercises: 


1. Create a list of integers. Use a for: statement to print out each integer in the list. 
2. Create a string. print out each character in the string. 
Solutions: 


1. The for: statement can iterate over the items in a list: 


Iba Abs s ey" |al  22 sisi | 
In [14]: for value in a: 
sughenis print 'value: %d' % value 
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2. The for: statement can iterate over the characters in a string: 


im hols ba chocolare: 
ier PLS score elaieil aliay Joye 
Saree joseabiaie, Ji@leelecicleaies asl a elaueil 


character: 
character: 
character: 
character: 
character: 
character: 
character: 
character: 
character: 


tar I= oi Oar @! 


Notes: 
o Inthe solution, I used the variable name chr1 rather than chr so as not to 
over-write the name of the built-in function chr (). 
When we need a sequential index, we can use the range () built-in function to create a 
list of integers. And, the xrange () built-in function produces an interator that produces 
a sequence of integers without creating the entire list. To iterate over a large sequence of 
integers, use xrange() instead of range (). 


Exercises: 


1. Print out the integers from 0 to 5 in sequence. 
2. Compute the sum of all the integers from 0 to 99999. 
3. Given the following generator function: 


abmereneic, sie iL Aas) 


Ueiliss =. )i 

Thttp://yahoo.icom", 

WinME Iea\2y// / eneiaveyal q(oncte] ! 

Vigheeiere 7 /Meatinjor,cenetes The GNU image manipulation 
program 


] 


olese wells (uaclL asic) 2 
ene bell alien hell kaise. 8 
Uren aclkopen (ui) 
stuff = f.read() 
f.close() 
yield stuff 


Write a for: statement that uses this iterator generator to print the lengths of the 
content at each of the Web pages in that list. 
Solutions: 


1. The range () built-in function gives us a sequence to iterate over: 
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iew oy) o aeoe “alieb:< aio seeiave rs) {(N6))) 2 
johenniones Vache seh! 25 aiiob:< 


alo bse 
alge bea 
abs 
iilnelecar: 
eelscs 
aLOb<e 


Gr iS 1) ier ea oo 


2. Since that sequence is a bit large, we'll use xrange () instead of range (): 


bey [Ss eon = 10) 
Asay SEB aeoe iol aay erento iS (ALON) 3 
8 CoUme ys ii 


ay abl) a" serosa 
Oe [ALO s AVI GsOoow 


3. The for: statement enables us to iterate over iterables as well as collections: 


ajjerenaie, ible ih lato) 


Uris — | 

Uinta ea/syclroome ons, 

Wisietejers // /fenmelaveva (oucio)! 

Wishes / // cabs cnaey! y The GNU image manipulation 
program 


] 


lene welll < (bell Iai sie) 2 
imene Well alien waedle Wats. 3 
12 = Phe llilalloy, ule ley ereiny (fuel) 
stuff = f.read() 
f.close() 
yield stuff 


def test(): 
hot Waele acm, walks (Uicilic)) 7 
print “length: <d' = (len(url), ) 


aL ie name == '  main__': 
test () 


When I ran this script, it prints the following: 


length: 9562 
length: 16341 
length: 12343 


If you need an index while iterating over a sequence, consider using the enumerate () 
built-in function. 
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Exercises: 


1. Given the following two lists of integers of the same length: 


an = (eee 2y By 4, >| 
l= [MOS 2007 SOO > Awe a0) 


Add the values in the first list to the corresponding values in the second list. 
Solutions: 


1. The enumerate () built-in function gives us an index and values from a 
sequence. Since enumerate () gives us an interator that produces a sequence of 
two-tuples, we can unpack those tuples into index and value variables in the 
header line of the for statement: 


ial Sil et =" [Ie 2 3 A 

La Lae b= OO 2ZOCr S007 200re S00) 

Tere [ele Sac: 

In [16]: for idx, value in enumerate(a): 
sae isvans b[idx] += value 

iey |by ls Ish 

Ona) laa OM Ore OS ed Os S10) oul 


3.5.5 while: statement exercises 


A while: statement executes a block of code repeatedly as long as a condition is true. 


Here is a template for the while: statement: 


while condition: 
statement 
fe) 
fe) 
fe) 


Where: 


e condition is an expression. The expression is something that returns a value 
which can be interpreted as true or false. 
Exercises: 


1. Write awhile: loop that doubles all the values in a list of integers. 
Solutions: 


1. Awhile: loop with an index variable can be used to modify each element of a 
list: 


def test_while(): 
numbers = [11, 22, 33, 44, ] 
print ‘before: %s' % (numbers, ) 
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idx = 0 

while idx < len(numbers): 
numbers [idx] *= 2 
idx += 1 

Print “after: t¢s' 3 (numbers, ) 


But, notice that this task is easier using the for: statement and the built-in 
enumerate () function: 


def test_for(): 
numbers = [11, 22, 33, 44, ] 
print 'before: %s' % (numbers, ) 
for idx, item in enumerate (numbers): 
numbers [idx] *= 2 
print ‘after: %s' % (numbers, ) 


3.5.6 break and continue statements 


The continue statement skips the remainder of the statements in the body of a loop 
and starts immediately at the top of the loop again. 


A break statement in the body of a loop terminates the loop. It exits from the 
immediately containing loop. 


break and continue can be used in both for: and while: statements. 
Exercises: 


1. Write a for: loop that takes a list of integers and triples each integer that is even. 
Use the cont inue statement. 
2. Write a loop that takes a list of integers and computes the sum of all the integers 
up until a zero is found in the list. Use the break statement. 
Solutions: 


1. The continue statement enables us to "skip" items that satisfy a condition or 
test: 


def test(): 
numbers = il, 22, 33, 44, 55, 66, | 
print ‘before: %s' % (numbers, ) 
for idx, item in enumerate (numbers): 
if item $ 2 != 0: 
continue 
numbers [idx] *= 
joucalinue Weneweies isi! 


S 
(numbers, ) 


test () 


2. The break statement enables us to exit from a loop when we find a zero: 


def test(): 
numbers — [lil 227 35, 0,7 44, 55, 66, | 
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° 


print ‘numbers: %s' % (numbers, ) 
sum = 0 
for item in numbers: 
if item == 
break 
sum += item 
PLeink “sumie sd) so (sum, |) 


test () 


3.5.7 Exceptions and the try:except: and raise statements 


The try:except: statement enables us to catch an exception that is thrown from 
within a block of code, or from code called from any depth withing that block. 


The raise statement enables us to throw an exception. 


An exception is a class or an instance of an exception class. If an exception is not caught, 
it results in a traceback and termination of the program. 


There is a set of standard exceptions. You can learn about them here: Built-in Exceptions 
-- http://docs.python.org/lib/module-exceptions.html. 


You can define your own exception classes. To do so, create an empty subclass of the 
class Exception. Defining your own exception will enable you (or others) to throw 
and then catch that specific exception type while ignore others exceptions. 


Exercises: 


1. Writeatry:except: statement that attempts to open a file for reading and 
catches the exception thrown when the file does not exist. 
Question: How do you find out the name of the exception that is thrown for an 
input/output error such as the failure to open a file? 

2. Define an exception class. Then write a try:except: statement in which you 
throw and catch that specific exception. 

3. Define an exception class and use it to implement a multi-level break from an 
inner loop, by-passing an outer loop. 

Solutions: 


1. Use the Python interactive interpreter to learn the exception type thrown when a 
1/O error occurs. Example: 


>>> infile = open('xx_nothing__yy.txt', 'r"') 
Traceback (most recent call last): 

nih le “VxseelbosW stig: ily shia, <oaerch tiles: 
LOPGrOr<) [Rercno 2) No vsuch tile Vor directory: 
oe ovoelenliave Aye ce 
= 
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In this case, the exception type is 1 


[OMe O 


Now, write a try:except: block which catches that exception: 


def test(): 

infilename 
easivas 

aliquip al] 

BONE 


"nothing_noplace.txt' 


le = open(infilename, 'r') 
daisavey alfal alialaeal ilreye 

print line 
erdceione WOlueiioue, iSe<er8 


print ‘cannot open file 


"' os infilename 


Wo 
oS 


test () 


2. We define a exception class as a sub-class of class Exception, then throw it 
(with the raise statement) and catch it (witha try:except: statement): 


class SizeError (Exception): 
pass 


def test_exception(size): 
Bases 
if size <= 0: 
raise SizeError, 
zero' 


Produce a different error to show that it 
will not be caught. 
x ay 
except Sizel exp: 
joueatione Urs, (exp, 
print 'goodbye' 


Huciaroes, 


° 
ix) 


) 


def test(): 
test_exception (-1) 
PEIN SS - 740 
test_exception (1) 


test () 


When we run this script, it produces the following output: 


$ python workbook027.py 
size must be greater than zero 


goodbye 
Mecicelocicls (iulosie wSicrsine eelllil Ibesic)) 3 
File "workbook027.py", line 20, in <module> 
cesic 
File "workbook027.py", line 18, in test 
test_exception (1) 
File "workbook027.py", line 10, in test_exception 
eae LY. 
NameError: global name 'y' is not defined 


Page 208 


"size must be greater than 


A Python Book 


Notes: 
o Our except: clause caught the SizeError, but allowed the NameError 
to be uncaught. 
3. We define a sub-class of of class Exception, then raise it in an inner loop and 
catch it outside of an outer loop: 


class BreakExceptionl (Exception): 


pass 
def test(): 
ay ==) (Mai 2 83,7 447 too, “Oo | 
b= ida 2227, S33, 444, S55, 666, | 
Tenens 
ieee alin) Ele 


joneninahes  Noibhereiie FT o:44) seach axe 
He@ne yf aligl loys 
if x > 22 and y > 444: 
okie BEcakixcepe Lomi ( koa nag 


inner loop') 


pieinte Samnerm S73 esd =cn ny 
PLIne Voukem iS idiswemy 
Deimey Lots 40 
xcept BreakExceptionl, exp: 
print “out of loop —- exp: ts" % exp 


test () 


Here is what this prints out when run: 


rhe Gye <5 
nlsguavene = 23 ALAlAL 
LNG — = yan ee 
TMM = — ven SS 
inner -- y: 444 
alliguayenic = ——9 Ais) s)0) 
inner -- y: 666 
Quleeie == Eciisie 
Oubero—— ae 22 
aljguevenc = 469) AL AnAL 
aLiQuOeNe = Ae 2A 
TINE —— SS 
inner -- y: 444 
ASTIN AN = Vat COD) 
alone === i) GiGl6) 
Owls =—— Elsie 
Ouver = — 2esnss 
AMC yan lhl 
UMNee = yee 
LAME == a 38S 
inner -- y: 444 
out of loop -—- exp: leaving inner loop 
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3.6 Functions 
A function has these characteristics: 


It groups a block of code together so that we can call it by name. 

It enables us to pass values into the the function when we call it. 

It can returns a value (even if None). 

When a function is called, it has its own namespace. Variables in the function are 
local to the function (and disappear when the function exits). 

A function is defined with the def: statement. Here is a simple example/template: 


def function_name(argl, arg2): 


Owed seiel = eiegil a il 
No @alljar2 —— aia. eZ, 
“return Local ivarl + localivar2 


And, here is an example of calling this function: 


result = function_name(1, 2) 


Here are a few notes of explanation: 


e The above defines a function whose name is function_name. 

e The function function_name has two arguments. That means that we can and 
must pass in exactly two values when we call it. 

e This function has two local variables, Local_varl and local_var2. These 
variables are local in the sense that after we call this function, these two variables 
are not available in the location of the caller. 

e When we call this function, it returns one value, specifically the sum of 
local_varl and local_var2. 

Exercises: 


1. Write a function that takes a list of integers as an argument, and returns the sum 
of the integers in that list. 
Solutions: 


1. The return statement enables us to return a value from a function: 


def list_sum(values): 
sum = 0 
for value in values: 
sum += value 
return sum 


def test (): 
a= (lik, 22, 33, 44, 1 
print list_sum(a) 


aie name == '  main__': 
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test () 


3.6.1 Optional arguments and default values 
You can provide a default value for an argument to a function. 

If you do, that argument is optional (when the function is called). 
Here are a few things to learn about optional arguments: 


e Provide a default value with an equal sign and a value. Example: 


def sample_func(argl, arg2, arg3='empty', arg4=0): 


e All parameters with default values must be after (to the right of) normal 
parameters. 

e Do not use a mutable object as a default value. Because the def: statement is 
evaluated only once and not each time the function is called, the mutable object 
might be shared across multiple calls to the function. Do not do this: 


def sample_func(argl, arg2=[]): 


Instead, do this: 


def sample_func(argl, arg2=None) : 
if arg2 is None: 
arg2 = [] 


Here is an example that illustrates how this might go wrong: 


def adder(a, b=[]): 
b.append (a) 
return b 


Clene ieesic (() 2 
print adder('aaa') 
joueulioe: elelelene ( olels\ )) 
Dent iaddersiGicce) 


test () 


Which, when executed, displays the following: 


['aaa'] 
[Yee Volos)" || 
Vas Vella! “ece! | 


Exercises: 


1. Write a function that writes a string to a file. The function takes two arguments: 
(1) a file that is open for output and (2) a string. Give the second argument (the 
string) a default value so that when the second argument is omitted, an empty, 
blank line is written to the file. 
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2. Write a function that takes the following arguments: (1) a name, (2) a value, and 
(3) and optional dictionary. The function adds the value to the dictionary using the 
name as a key in the dictionary. 
Solutions: 


1. We can pass a file as we would any other object. And, we can use a newline 
character as a default parameter value: 


import sys 


def writer(outfile, msg='\n'): 
outfile.write (msg) 


clase jesse () 8 
writer(sys.stdout, 'aaaaa\n') 
writer (sys.stdout) 
writer(sys.stdout, 'bbbbb\n') 


test () 


When run from the command line, this prints out the following: 


aaaaa 


bbbbb 


2. In this solution we are careful not to use a mutable object as a default value: 


def add_to_dict (name, value, dic=None): 
alge Glalieh aks) iNKouaes 
dic = {} 
dic[name] = value 
wee when Clhye 


def test(): 
Givel = {Valiberis “eure, } 
joncalione, ciclo wen ichiere ((Viecneiay 5) Vaeuhauey tela) 


( 

prin’ ~addltordict ("charlene “smant"’, dic) 
joueabiaie, ciclo mec Chics (( heksieiay Is Vroybhescereievoubls )) 
print add_to_dict('eddie', 'friendly') 

test () 

If we run this script, we see: 

(Ubacey =: “Siunmny , “ocllbere es" Neuter} 

i"barny "> “tunny", “allberi’: “cute, *charlene': 

Wiser! jy 

1 "darryd' = “Voutmageous | 

{'eddie': 'friendly'} 


Notes: 
o It's important that the default value for the dictionary is None rather than an 
empty dictionary, for example ({ }). Remember that the def: statement is 
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evaluated only once, which results in a single dictionary, which would be 
shared by all callers that do not provide a dictionary as an argument. 


3.6.2 Passing functions as arguments 


A function, like any other object, can be passed as an argument to a function. This is due 
the the fact that almost all (maybe all) objects in Python are "first class objects". A first 
class object is one which we can: 


1. Store in a data structure (e.g. a list, a dictionary, ...). 
2. Pass to a function. 
3. Return from a function. 

Exercises: 


1. Write a function that takes three arguments: (1) an input file, (2) an output file, 
and (3) a filter function: 
o Argument | is a file opened for reading. 
o Argument 2 is a file opened for writing. 
o Argument 3 is a function that takes a single argument (a string), performs a 

transformation on that string, and returns the transformed string. 
The above function should read each line in the input text file, pass that line 
through the filter function, then write that (possibly) transformed line to the 
output file. 
Now, write one or more "filter functions" that can be passed to the function 
described above. 
Solutions: 


1. This script adds or removes comment characters to the lines of a file: 


import sys 


lene aealinee (alinizai ike. toihescal ike, sestaliceycneibhaye))) 4 
For lainey alin asa ler 

line = filterfunc (line) 

outfile.write (line) 


def add_comment (line): 
line = '## %s' % (line, ) 
return line 


def remove_comment (line): 
if line.startswith('## '): 
line = line[3:] 
return line 


def main(): 
filter(sys.stdin, sys.stdout, add_comment) 
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aL ie name == '  main__': 
main () 


Running this might produce something like the following (note for MS Windows 
users: use type instead of cat): 


8 Cele jello ioe 
line 1 
line 2 
line 3 
S Gat tmp txt ||| python workbook005 py 
## line 1 
line 2 
ine 3 


3.6.3 Extra args and keyword args 


Additional positional arguments passed to a function that are not specified in the function 
definition (the def: statement™’), are collected in an argument preceded by a single 
asterisk. Keyword arguments passed to a function that are not specified in the function 
definition can be collected in a dictionary and passed to an argument preceded by a 
double asterisk. 


Examples: 


1. Write a function that takes one positional argument, one argument with a default 
value, and also extra args and keyword args. 
2. Write a function that passes all its arguments, no matter how many, to a call to 
another function. 
Solutions: 


1. We use *args and **kwargs to collect extra arguments and extra keyword 
arguments: 


def show_args(x, y=-l1, *args, **kwargs): 
TONe Anite te, EAN) 
joneniiohen sUoeG Ee 
jouealions Vag! yy 
joneabione, | Uehereisyn ll hac is: 
print 'kwargs:', kwargs 


def test(): 


show_args 


( 
( 
show_args ( 
( 
show_args ( 


test () 
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Running this script produces the following: 


S python workbook006.py 


Vien 
suseise (Si, Wa 33) 
kwargs: {} 


ocese Salle 

y: 44 

aregse 0) 

kwargs: vals Sid, Vbls Got 


Notes: 
o The spelling of args and kwargs is not fixed, but the 
2. Weuse args and kwargs to catch and pass on all arguments: 


det sume (sors; 7 Kkwanois) = 


fo) 


joueiinje, Veneeisg EY a (enecisi, )) 


jouerioie, Vieiwenecisg fais! ay (icles, 
det funce2 (*args, **kwargs) : 

print 'before' 

SUM (SAAC Sia Wicl ta Chss) 

joueations  Valae cere | 


def test(): 
tune? (Yaaa', “bbb, “cece, argil—"“ddd", arg2z—"6eeE") 


test () 


When we run this, it prints the following: 


before 

suceise (Veleley 4 Vielola! = Vetere!) 

Kwangs) (Vanoql s Vadd" 7) “ang2z" i: Nece™ } 
after 
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Notes: 

o Ina function call, the * operator unrolls a list into individual positional 
arguments, and the ** operator unrolls a dictionary into individual keyword 
arguments. 


3.6.3.1 Order of arguments (positional, extra, and keyword args) 
In a function definition, arguments must appear in the following order, from left to right: 


1. Positional (normal, plain) arguments 
2. Arguments with default values, if any 
3. Extra arguments parameter (proceded by single asterisk), if present 
4. Keyword arguments parameter (proceded by double asterisk), if present 
In a function call, arguments must appear in the following order, from left to right: 


1. Positional (plain) arguments 
2. Extra arguments, if present 
3. Keyword arguments, if present 


3.6.4 Functions and duck-typing and polymorphism 


If the arguments and return value of a function satisfy some description, then we can say 
that the function is polymorphic with respect to that description. 


If the some of the methods of an object satisfy some description, then we can say that the 
object is polymorphic with respect to that description. 


Basically, what this does is to enable us to use a function or an object anywhere that 
function satisfies the requirements given by a description. 


Exercises: 


1. Implement a function that takes two arguments: a function and an object. It 
applies the function argument to the object. 
2. Implement a function that takes two arguments: a list of functions and an object. 
It applies each function in the list to the argument. 
Solutions: 


1. We can pass a function as an argument to a function: 


def fancy(obj): 
joueaions Vaesiney scein@\yy ——" fas = iceyarely seeineyy) Ss (olag|, )) 


def plain(obj): 
jonesione,  “eyleisiign —— as} == olen 5 (Glog, 4) 


def show(func, obj): 
func (obj) 
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def main(): 
eye baie li Volo es aA 
show(fancy, a) 
show(plain, a) 


alae name == ' main __': 
main () 


2. We can also put functions (function objects) in a data structure (for example, a 
list), and then pass that data structure to a function: 


def fancy(obj): 


joneuiiote, Vieeiereyy, seciniGyy == fs) —— aeclaveyy irehoey) a (loa, )) 
den plaan (Oba): 

jouealioge, Wolleabe == sj) —— ellen Y S  lelogie )) 
Eunce isis —) |brancy, ) plantar || 


def show(funcs, obj): 
Oi Gewhale! alia) Sewbayelsse 
func (obj) 


def main(): 
a=] {t'aa"s lly, “bb's 22, + 
show(Func_list, a) 


alge name == '  main__': 
main () 


Notice that Python supports polymorphism (with or) without inheritance. This type of 
polymorphism is enabled by what is called duck-typing. For more on this see: Duck 
typing -- http://en.wikipedia.org/wiki/Duck_typing at Wikipedia. 


3.6.5 Recursive functions 
A recursive function is a function that calls itself. 
A recursive function must have a limiting condition, or else it will loop endlessly. 


Each recursive call consumes space on the function call stack. Therefore, the number of 
recursions must have some reasonable upper bound. 


Exercises: 


1. Write a recursive function that prints information about each node in the 
following tree-structure data structure: 


Tree = { 
“mame! Varaameaullis 
YAS lovecliavelaw ss 4 
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Uiovetine/ 8 © VM ouilieiosy! 7 
WILE igie wlewsciiavelaY Ss 
‘name': 'seed eaters', 
Nileieabirancin = ef 
“mame: Vnouse: fimeh!, 


‘left_branch': None, 
'right_branch': None, 
}, 
Venghteoramneina 24 
"name': 'white crowned sparrow', 
‘left_branch': None, 
‘right branch’: None, 
}, 
}, 
Wiese ioe Joiecuarela! Bl 
‘name': 'insect eaters', 
Vilierecbraneh | 
Vigvetine! 3 Visieueimulie ielavebisia! 5 
“lett branch's None; 
"right_branch': None, 
}, 
reat ince Jonechaclany | 
"name': 'black headed phoebe', 
"‘left_branch': None, 
Yrightobranch'=) None, 
}, 
}, 


}, 
‘right_branch': None, 


Solutions: 


1. We write a recursive function to walk the whole tree. The recursive function calls 
itself to process each child of a node in the tree: 


Tree = { 
‘mame "2 “animes, 
Vlettubranch ss “| 
Mota! 8 Voaliselss! 5 
ViketeEomaneh yy 
‘name': 'seed eaters', 
Viet Ee obiraneh i) =] 
Vigkeliles) &  Vinwoxbhsve: ealinvelay\’ - 


‘left_branch': None, 
"right_branch': None, 

}, 

scan ioe, lonacunelay As =| 
"name': 'white crowned sparrow', 
‘left_branch': None, 
"right_branch': None, 


}, 


}, 
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Ueak@ileie Jeneeherel 2 | 
"name': 'insect eaters', 
‘lett branch’) 7 
‘name’: Yhermit Ehrush!, 
‘left_branch': None, 
"right_branch': None, 
Ihe 
Uical@ lee lencebaveley! 2) I 
‘name’: 'black headed phoebe', 
‘left_branch': None, 
"right_branch': None, 
}y 
ty 
ia 
"right_branch': None, 


} 


Indents = [' Us Vakiob< azone aleb< aliay aerehove ey ((1l(0))) | 


def walk_and_show(node, level=0): 
if node is None: 


return 
print '%Ssname: %s' % (Indents[level], node['name'], 


level += 1 
walk_and_show(node['left_branch'], level) 
walk_and_show(node['right_branch'], level) 


def test(): 
walk_and_show (Tree) 


aie name == '  main_': 


test () 


Notes: 

o Later, you will learn how to create equivalent data structures using classes and 
OOP (object-oriented programming). For more on that see Recursive calls to 
methods in this document. 


3.6.6 Generators and iterators 


The "iterator protocol" defines what an iterator object must do in order to be usable in an 
"iterator context" such as a for statement. The iterator protocol is described in the 
standard library reference: Iterator Types -- http://docs.python.org/lib/typeiter.html 


An easy way to define an object that obeys the iterator protocol is to write a generator 
function. A generator function is a function that contains one or more yield statements. 
If a function contains at least one yield statement, then that function when called, 
returns generator iterator, which is an object that obeys the iterator protocol, i.e. it's an 
iterator object. 
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Note that in recent versions of Python, yield is an expression. This enables the consumer 
to communicate back with the producer (the generator iterator). For more on this, see 
PEP: 342 Coroutines via Enhanced Generators - 
http://www.python.org/dev/peps/pep-0342/. 


Exercises: 


1. Implement a generator function -- The generator produced should yield all 
values from a list/iterable that satisfy a predicate. It should apply the transforms 
before return each value. The function takes these arguments: 

1. values -- A list of values. Actually, it could be any iterable. 

2. predicate -- A function that takes a single argument, performs a test on 
that value, and returns True or False. 

3. transforms -- (optional) A list of functions. Apply each function in this list 
and returns the resulting value. So, for example, if the function is called like 


this: 


igeeibllicg = Weieeyorsnnonauis (bile 227 joe ser teh) 


then the resulting generator might return: 
ise (EE) ) 

2. Implement a generator function that takes a list of URLs as its argument and 
generates the contents of each Web page, one by one (that is, it produces a 
sequence of strings, the HTML page contents). 

Solutions: 


1. Here is the implementation of a function which contains yield, and, therefore, 
produces a generator: 


!/usr/bin/env python 


wu 


filter _and_transform 


Piltten and we rans KormlicConbeniy,  Eecica sume, 
transforms=None) 


Return a generator that returns items from content 


after applying 
the functions in transforms if the item satisfies 


test_func 


Me gumemr sss 

i. walties  =— A Vist of values 

2. ~ predicate ~ -- A function that takes a single 
argument, 


performs a test on that value, and returns True 
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or False. 


320° eeanstouns) = (opelonal) A list or functions. 
Apply each 
function in this list and returns the resulting 
welllles Se, 
for example, if the function is called like 
Phases: 


isesillic = aesllheeie cincl cece (i, 225 jo, see 
g]) 


then the resulting generator might return:: 


@p(Ge (AL) ) 


deh filter land transrorm(content, test fume, 
transforms=None) : 
FOr) s6 Sim omen: 
sie) (HSS Sebi! (52) 2 
if transforms is None: 
yield x 
elif isiterable(transforms): 
fom une an trans forme: 
x = func (x) 
yield x 
else: 
yield transforms (x) 


def isiterable(x): 
ie licie) =" Miele 
Gasyvas 
x = iter (x) 
xcs Or WyjoSlieicoie, esqoe 
lec = ellse 


return flag 


def iseven(n): 


° 


return n % 2 == 0 


lene aeqGal) 2 
sete eiay inp 7 2 


def g(n): 
mee atenacn ess a2 
def test (): 
datvall = iil, 227) 387 «44,, oo, OO, Fl iil 
ions Well ier teal leSie Vetere! 2 welaisuconciml (ovicelll, alseweiay, ab) 2 
jouesique, UNiehike Fael! Ss t(igstl 
TOA ey a ae eA) 
for val in filter_and_transform(datal, iseven, [f, 
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ill) 8 
print 'val: %d' % (val, ) 
jonenbigheee e710) 
Om Viele aim allibenrsand. ticametonem (Gait alley) assiexren)) ij: 
jouesume Velie acl’ ss (yall, )) 
aL ie name == '  main__': 
test () 


Notes: 

o Because function filter_and_transform contains yield, when 

called, it returns an iterator object, which we can use in a for statement. 

The second parameter of function £ilter_and_transfornm takes any 

function which takes a single argument and returns True or False. This is an 

example of polymorphism and "duck typing" (see Duck Typing -- 

http://en. wikipedia.org/wiki/Duck_typing). An analogous claim can be made 

about the third parameter. 

2. The following function uses the url1ib module and the yield function to 
generate the contents of a sequence of Web pages: 


almexeueie, sell Ia) 


Ureiks 


[ 
“http: //yahoo.com", 
WigeIejal3 //// enyieiavovare (onaie] 
Wistetcjer2 // /Mealiiaay, ence]! 
program 


] 


# The GNU image manipulation 


det walk (aie last): 
ene uel ale weed asic. 3 
if urllib.urlopen (url) 
SHC WIIE IE f.read() 


ooo 


lose () 


yiel 


def test(): 


ene oe alia aie 
joueaLione 


lich Siebieie 


Lic (Ufellss)) 3 
Nengteh: 


T 


main 


When I run this, I see: 


S python generator_example.py 


length: 9554 
length: 16748 
length: 11487 
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3.7 Object-oriented programming and classes 


Classes provide Python's way to define new data types and to do OOP (object-oriented 
programming). 


If you have made it this far, you have already used lots of objects. You have been a 
"consumer" of objects and their services. Now, you will learn how to define and 
implement new kinds of objects. You will become a "producer" of objects. You will 
define new classes and you will implement the capabilities (methods) of each new class. 


A class is defined with the class statement. The first line of a class statement is a 
header (it has a colon at the end), and it specifies the name of the class being defined and 
an (optional) superclass. And that header introduces a compound statement: specifically, 
the body of the class statement which contains indented, nested statements, 
importantly, def statements that define the methods that can be called on instances of the 
objects implemented by this class. 


Exercises: 


1. Define a class with one method show. That method should print out "Hello". 
Then, create an instance of your class, and call the show method. 
Solutions: 


1. A simple instance method can have the self parameter and no others: 


class Demo(object): 
def show(self): 
Disimt vine Tike 


def test(): 
a = Demo () 
a.show() 


test () 


Notes: 

o Notice that we use object as a superclass, because we want to define an 
"new-style" class and because there is no other class that we want as a 
superclass. See the following for more information on new-style classes: 
New-style Classes -- http://www.python.org/doc/newstyle/. 

o In Python, we create an instance of a class by calling the class, that is, we 
apply the function call operator (parentheses) to the class. 


3.7.1. The constructor 


A class can define methods with special names. You have seem some of these before. 
These names begin and end with a double underscore. 
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One important special name is___init__. It's the constructor for a class. It is called 
each time an instance of the class is created. Implementing this method in a class gives us 
a chance to initialize each instance of our class. 


Exercises: 


1. 


Implement a class named Plant that has a constructor which initializes two 
instance variables: name and size. Also, in this class, implement a method 
named show that prints out the values of these instance variables. Create several 
instances of your class and "show" them. 

Implement a class name Node that has two instance variables: data and 
children, where data is any, arbitrary object and children is a list of child 
Nodes. Also implement a method named show that recursively displays the 
nodes in a "tree". Create an instance of your class that contains several child 
instances of your class. Call the show method on the root (top most) object to 
show the tree. 


Solutions: 


1. 


The constructor for a class is a method with the special name ___init__: 


class Plant (object): 
Ce 2 nuiicn (Selah mame sesame 
self.name = name 
self.size = size 
def show(self): 
print ‘name: "Ss" size: %Sd' % (self.name, 
self.size, ) 


def test(): 
jul = 2 ikenae (inejejolleiene Ye 25) 
D2 — Pilani Gonmarot; 36) 
jollcincs = joll, jaz, | 
hor pilent “in plants: 
plant.show() 


test () 


Notes: 

o Our constructor takes two arguments: name and size. It saves those two 
values as instance variables, that is in attributes of the instance. 

o The show() method prints out the value of those two instance variables. 

It is a good idea to initialize all instance variables in the constructor. That enables 

someone reading our code to learn about all the instance variables of a class by 

looking in a single location: 


# simple_node.py 


Indents = [' ' * n for n in range(10) ] 
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class Node(object): 
def __ init__(self, name=None, children=None): 
self.name = name 
it Chanliciwen sas Nonet 
self.children = [] 


else: 
self.children = children 
def show_name(self, indent): 
print 'Ssname: "Ss"' % (Indents[indent], 


self.name, ) 
def show(self, indent=0): 
self.show_name (indent) 
indent += 1 
eves felosillel: alay iioulan roloallicketyaye 
child. show (indent) 


def test(): 
nl = Node('N1') 
n2 = Node('N2"') 
n3 = Node('N3') 
n4 = Node('N4') 
no — Node({Ns", [mil, m2, 1) 
moe NoderCUNG! 7.) [asym |.) 
n7 = Node('N7'! Marsrs. sailsr ll) 
n7.show () 

aie name == ' main __': 
test () 


Notes: 

o Notice that we do not use the constructor for a list ([ ] ) as a default value for 
the children parameter of the constructor. A list is mutable and would be 
created only once (when the class statement is executed) and would be shared. 


3.7.2 Inheritance -- Implementing a subclass 


A subclass extends or specializes a superclass by adding additional methods to the 
superclass and by overriding methods (with the same name) that already exist in the 
superclass. 


Exercises: 


1. Extend your Node exercise above by adding two additional subclasses of the 
Node class, one named Plant and the other named Animal. The Plant class 
also has a height instance variable and the Animal class also has a color 
instance variable. 

Solutions: 


1. Wecan import our previous Node script, then implement classes that have the 
Node class as a superclass: 


Page 225 


A Python Book 


from simple_node import Node, Indents 


class Plant (Node): 
def __init__(self, name, height=-1, children=None): 
Node.__init__ (self, name, children) 
self.height = height 
def show(self, indent=0): 
self.show_name (indent) 
print 'Ssheight: %s' % (Indents[indent], 
self.height, ) 
indent += 1 
hor sehuslice in sesh woh slkeieen: 
child.show (indent) 


class Animal (Node): 


dekh Lint. selkk mame, CGolonr= "mo @odlorm', 
children=None): 
Node.__init__(self, name, children) 
Selft.color = icollor 


def show(self, indent=0): 
self.show_name (indent) 


pieing “casicollors “sc 4). (indents indent, 
Selec oOdlorss. 4) 
indent += 1 


for child ain seit sehiitdrent: 
child.show (indent) 


def test(): 

nl = Animal ('scrubjay', 'gray blue") 

n2 = Animal ('raven', "ollack”) 

n3 = Animal('american kestrel', '‘brown') 

n4 = Animal ('red-shouldered hawk', 'brown and 
gray") 

ido) =" Naum (Aneorewarel Visions lial ia247 ll) 

n6é = Animal ('raptor', children=[n3, n4,]) 


mea —) Anameade( Vbstcd)) vochiviidisen—imSy. jmGi. ||) 
rays Plant ('valley oak', 50) 

n2 = Plant('canyon live oak', 40) 

n3 = Plant ("jeffery pine', 120) 

n4 = Plant ('ponderosa pine', 140) 

no = Pilani (toak’) ichaikcren— milly maz ||) 

mo — Pilanti(veonimezer”) Tehakdtenm— lms, mAy i) 
n7vb = Plant ('tree', children=[n5, n6,]) 
n8 = Node('birds and trees', [n7a, n7b,]) 
n8.show() 


Ba Ee 


Notes: 
o The show method in class Plant calls the show_name method in its 
superclass using self.show_name(...). Python searches up the 
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inheritance tree to find the show_name method in class Node. 

o The constructor (__init__) inclasses Plant and Animal each call the 
constructor in the superclass by using the name of the superclass. Why the 
difference? Because, if (in the Plant class, for example) it used 
self.__init___(...) it would be calling the __init__ inthe Plant 
class, itself. So, it bypasses itself by referencing the constructor in the 
superclass directly. 

o This exercise also demonstrates "polymorphism" -- The show method is 
called a number of times, but which implementation executes depends on 
which instance it is called on. Calling on the show method on an instance of 
class Plant results ina call to Plant . show. Calling the show method on 
an instance of class Animal results in a call to Animal .show. And so on. It 
is important that each show method takes the correct number of arguments. 


3.7.3 Classes and polymorphism 


Python also supports class-based polymorphism, which was, by the way, demonstrated in 
the previous example. 


Exercises: 


1. Write three classes, each of which implement a show () method that takes one 
argument, a string. The show method should print out the name of the class and 
the message. Then create a list of instances and call the show () method on each 
object in the list. 

Solution: 


1. We implement three simple classes and then create a list of instances of these 
classes: 


class A(object): 
def show(self, msg): 
jouenine Vieilles: 4 —— iiisier? 


class B(object): 
def show(self, msg): 
joneabione Vichiieiss 13) == iste, h Wea Ee Giisicfy ) 


class C(object): 
def show(self, msg): 
joucalione Vieilles, 1 —— ise. Vist! as anise) 


Glee cesic () 3 
Oss = UN; E07 CO, B07 | 
for idx, obj in enumerate(objs): 
msg = 'messag Cl % (aiebs qr i 
ob j.show (msg) 
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Notes: 

o Wecan call the show () method in any object in the list ob js as long as we 
pass in a single parameter, that is, as long as we obey the requirements of 
duck-typing. We can do this because all objects in that list implement a 
show () method. 

o Ina statically typed language, that is a language where the type is (also) 
present in the variable, all the instances in example would have to descend 
from a common superclass and that superclass would have to implement a 
show () method. Python does not impose this restriction. And, because 
variables are not not typed in Python, perhaps that would not even possible. 

o Notice that this example of polymorphism works even though these three 
classes (A, B, and C) are not related (for example, in a class hierarchy). All 
that is required for polymorphism to work in Python is for the method names 
to be the same and the arguments to be compatible. 


3.7.4 Recursive calls to methods 


A method in a class can recusively call itself. This is very similar to the way in which we 
implemented recursive functions -- see: Recursive functions. 


Exercises: 


1. Re-implement the binary tree of animals and birds described in Recursive 
functions, but this time, use a class to represent each node in the tree. 
2. Solve the same problem, but this time implement a tree in which each node can 
have any number of children (rather than exactly 2 children). 
Solutions: 


1. We implement a class with three instance variables: (1) name, (2) left branch, and 
(3) right branch. Then, we implement a show () method that displays the name 
and calls itself to show the children in each sub-tree: 


Indents = [' Y * “idk hom ids ian range (10) | 


class AnimalNode (object): 


def __ init__(self, name, left_branch=None, 
right_branch=None) : 
self.name = name 


Selle, LSte iosaingl = iSite _leneeiniela 
self.right_branch = right_branch 


def show(self, level=0): 


° 


print 'Ssname: %s' % (Indents[level], 
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self.name, ) 
level += 1 
if self.left_branch is not None: 
self.left_branch.show(level) 
if self.right_branch is not None: 
self.right_branch.show (level) 


Tree = AnimalNode('animals', 
AnimalNode('birds', 

AnimalNode('seed eaters', 
AnimalNode('house finch'), 
AnimalNode('white crowned sparrow'), 

), 

AnimalNode('insect eaters', 
AnimalNode('hermit thrush'), 
AnimalNode('black headed phoebe'), 


)y 
)y 
None, 


) 


def test(): 
Tree. show() 


ange name == '  main__': 
test () 


2. Instead of using a left branch and a right branch, in this solution we use a list to 
represent the children of a node: 


class AnimalNode (object): 
Gethin GSelek daze, Clhunlidaeen—Nomere: 
self.data = data 
if children is None: 
self.children = [] 
else: 
sie lie , eloal lehaein 


(elavall Ieliaienal 


def show(self, level=''): 
print '%Ssdata: %s' % (level, self.data, ) 
level += '' i 
ime) elaal Ike! Maliak Seulae 4 eloval Weliseiar 
child.show (level) 


Tree = AnimalNode('animals', [ 
AnimalNode (‘birds', | 

AnimalNode('seed eaters', [ 
AnimalNode('house finch'), 
AnimalNode('white crowned sparrow'), 
AnimalNode('lesser gold finch"), 

]), 

AnimalNode('insect eaters', [ 
AnimalNode('hermit thrush'), 
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AnimalNode('black headed phoebe'), 
Ihe 
]) 
I) 


def test(): 
Tree. show() 


ale name == '  main__': 
este) 
Notes: 
o We represent the children of a node as a list. Each node "has-a" list of 
children. 


o Notice that because a list is mutable, we do not use a list constructor ([ ] ) in 
the initializer of the method header. Instead, we use None, then construct an 
empty list in the body of the method if necessary. See section Optional 
arguments and default values for more on this. 

o We (recursively) call the show method for each node in the children list. 
Since a node which has no children (a leaf node) will have an empty 
children list, this provides a limit condition for our recursion. 


3.7.5 Class variables, class methods, and static methods 


A class variable is one whose single value is shared by all instances of the class and, in 
fact, is shared by all who have access to the class (object). 


"Normal" methods are instance methods. An instance method receives the instance as its 
first argument. A instance method is defined by using the def statement in the body of a 
class statement. 


A class method receives the class as its first argument. A class method is defined by 
defining a normal/instance method, then using the classmethod built-in function. For 
example: 


class ASimpleClass (object): 
description = 'a simple class' 
def show_class(cls, msg): 


joes Ves ssl 5 (ClsicesemijocoM , mse, )) 
show_class = classmethod(show_class) 


A static method does not receive anything special as its first argument. A static method is 
defined by defining a normal/instance method, then using the stat icmethod built-in 
function. For example: 


class ASimpleClass (object): 
description = 'a simple class' 
def show_class (msg): 
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° 


print '%Ss: %Ss' % (ASimpleClass.description , msg, ) 
show_class = staticmethod(show_class) 


In effect, both class methods and static methods are defined by creating a normal 
(instance) method, then creating a wrapper object (a class method or static method) using 
the classmethod or staticmethod built-in function. 


Exercises: 


1. Implement a class that keeps a running total of the number of instances created. 
2. Implement another solution to the same problem (a class that keeps a running 
total of the number of instances), but this time use a static method instead of a 
class method. 
Solutions: 


1. We use aclass variable named instance_count, rather than an instance 
variable, to keep a running total of instances. Then, we increment that variable 
each time an instance is created: 


class CountInstances (object): 


instance_count = 0 
def abigubie (self, name='-no name-'): 
self.name = name 


CountInstances.instance_count += 1 


def show(self): 
print 'name: "%s"' % (self.name, ) 


def show_instance_count (cls): 
print ‘instance count: %d' % 
(cls.instance_count, ) 
show_instance_count = 
classmethod (show_instance_count) 


cle esse () Ss 
instances = [] 
instances.append(CountInstances ('apple') ) 
instances.append(CountInstances ('banana') ) 
instances.append(CountInstances('cherry') ) 
( 


)) 


instances.append(CountInstances 

for instance in instances: 
instance.show() 

CountInstances.show_instance_count () 


aie name == '  main_': 


Notes: 
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o When we run this script, it prints out the following: 


name: "apple" 
name: "banana" 
ovens Wielaveieieyy 
name: "-no name-" 
instance count: 4 


o The call to the classmethod built-in function effectively wraps the 
show_instance_count method in a class method, that is, in a method 
that takes a class object as its first argument rather than an instance object. To 
read more about classmethod, go to Built-in Functions -- 
http://docs.python.org/lib/built-in-funcs.html and search for "classmethod". 

2. A static method takes neither an instance (self) nor a class as its first 
paramenter. And, static method is created with the staticmethod() built-in 
function (rather than with the classmethod () built-in): 


class CountInstances (object): 


instance_count = 0 
def alinyal te, (self, name='-no name-'): 
self.name = name 


CountInstances.instance_count += 1 


def show(self): 
print 'name: "%s"' % (self.name, ) 


def show_instance_count(): 
Prine “imstance count: 3d* < ( 
CountInstances.instance_count, ) 
show_instance_count = 
staticmethod(show_instance_count) 


Clee eesic () 2 
instances = [] 
instances.append(CountInstances ('apple') ) 
instances.append(CountInstances ('banana') ) 
instances.append(CountInstances('cherry') ) 
( 


instances.append(CountInstances () ) 
for instance in instances: 
instance. show () 


CountInstances.show_instance_count () 


aL ie name == '  main__': 
test () 


3.7.5.1 Decorators for classmethod and staticmethod 


A decorator enables us to do what we did in the previous example with a somewhat 
simpler syntax. 
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For simple cases, the decorator syntax enables us to do this: 


@functionwrapper 
def methodl (self): 
fe) 
fe) 
fe) 


instead of this: 


Clene iilsicloverell (Gsreilie)) = 
fe) 
© 
fe) 
methodl = functionwrapper (method1) 


So, we can write this: 


@classmethod 

def methodl (self): 
fe) 
fe) 
fe) 


instead of this: 


def methodl (self): 
fe) 
fe) 
fe) 
methodl = classmethod(method1) 


Exercises: 


1. Implement the Count Instances example above, but use a decorator rather 
than the explicit call to classmethod. 
Solutions: 


1. A decorator is an easier and cleaner way to define a class method (or a static 
method): 


class CountInstances (object): 


instance_count = 0 
def abiguiite (self, name='-no name-'): 
self.name = name 


CountInstances.instance_count += 1 


def show(self): 
print 'name: "%s"' % (self.name, ) 


@classmethod 
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def show_instance_count (cls): 
print ‘instance count: %d' % 
(cls.instance_count, ) 
# Note that the following line has been replaced by 
# the classmethod decorator, above. 
# show_instance_count = 
classmethod(show_instance_count) 


Glee ieee ()) 2 
instances = [] 
instances.append(CountInstances ('apple') ) 
instances.append(CountInstances ('banana') ) 
instances.append(CountInstances('cherry') ) 
( 


instances.append(CountInstances 

for instance in instances: 
instance.show() 

CountInstances.show_instance_count () 


oD) 


aL ie name == ' main __': 
test () 


3.8 Additional and Advanced Topics 


3.8.1 Decorators and how to implement them 
Decorators can be used to "wrap" a function with another function. 


When implementing a decorator, it is helpful to remember that the following decorator 
application: 


@dec 
def func(argl, arg2): 
pass 


is equivalent to: 


def func(argl, arg2): 
pass 
func = dec(func) 


Therefore, to implement a decorator, we write a function that returns a function object, 
since we replace the value originally bound to the function with this new function object. 
It may be helpful to take the view that we are creating a function that is a wrapper for the 
original function. 


Exercises: 


1. Write a decorator that writes a message before and after executing a function. 
Solutions: 
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1. A function that contains and returns an inner function can be used to wrap a 
function: 


def trace(func): 
def inner(*args, **kwargs): 
joucaioie. Ys 
EUMC Gee GS, | oo kwalacs) 
jeanne Val 
Ket UieM ssMaieie 


@trace 

def funcl(x, y): 
jouer, se see Meets ay 
EuMeZN( (a7, 37) 


@trace 


death saumezn(Comiecmity in: 
Ppeint “contents”, content 


def test(): 
seqalinnell(( eiei) 4 Noylc)'})) 


test () 


Notes: 
o Your inner function can use *args and **kwargs to enable it to call 
functions with any number of arguments. 
3.8.1.1 Decorators with arguments 
Decorators can also take arguments. 


The following decorator with arguments: 


@dec(argA, argB) 
der funeitangil, “anq2)): 
pass 


is equivalent to: 


der tune (arg, arg? ji: 
pass 
func = dec(argA, argB) (func) 


Because the decorator's arguments are passed to the result of calling the decorator on the 
decorated function, you may find it useful to implement a decorator with arguments using 
a function inside a function inside a function. 


Exercises: 


1. Write and test a decorator that takes one argument. The decorator prints a 
message along with the value of the argument before and after entering the 
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decorated function. 


Solutions: 


1. 


Implement this decorator that takes arguments with a function containing a nested 
function which in turn contains a nested function: 


def trace(msg) : 
lene “sidiareneil (Aeibleve!)) 8 
def inner2(*args, **kwargs): 


joiealigne Ves ess! ee iitsiey, 
retval = func(*args, **kwargs) 
pring “= sel' 3s Minso, ») 


return retval 
return inner2 
Rebun anne 


@trace('tracing funcl') 
det umcis Ger 37): 

oman, woes Ve se Mayen yy, 
maeCuikt — seme (x7. yn) 
return result 


@trace('tracing func2') 

def func2 (content): 
perme “contents 7. convent 
BebuGmMcontent. = Ss 


def test(): 
Besule =] funnel (aa; *bbt) 
jovcauoie  Viceebulhes Vo uessbulic 
test () 


3.8.1.2 Stacked decorators 


Decorators can be "stacked". 


The following stacked decorators: 


@dec2 
@decl 
Clee seule (eucc il, Bez, 266): 


pass 


are equivalent to: 


ClSis tebe (auaeil_ euce;A, 45 5) & 


pass 


func = dec2(decl (func) ) 


Exercises: 


1. 


Implement a decorator (as above) that traces calls to a decorated function. Then 
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Modify your solution to the above exercise so that the decorator that prints the 
horizontal line takes one argument: a character (or characters) that can be repeated 


to produce a horizontal line/separator. 


Solutions: 


Reuse your tracing function from the previous exercise, then write a simple 
decorator that prints a row of dashes: 


def trace(msg) : 
det amneied (tune) 
def inner2(*args, 
jonesliones Vs eh)! 
retval = func(*args, 
jouesuone VCs [asl Y 5 (Gisie) 7 
return retval 
return inner2 
weewhem siaieeie ll 


AGW alte 1S.) a 
% (msg, ) 
ASW clits chs!) 


) 


dekh horrvonteal Wime(fune)): 
def inner(*args, **kwargs): 
yonealiane, Vs oy) 
retval = func(*args, **kwargs) 
joneniighes a 5110) 


return retval 
KetuiRM ammete 


@trace('tracing funcl') 
det tunel, v)s 
joneaiohiey pecs es Zen c 
result imho (Ge 
return result 


7 


“GM 
y)) 


ChiorsisZiomt allele 

GCErace(TEracing £une2*) 

def func2 (content): 
pene “contents, 
return content * 3 


content 


def test(): 
result 
joueaLione 


UMC (Malas blo») 
Sresuilpe” ~ sees ult 


test () 


2. Once again, a decorator with arguments can be implemented with a function 
nested inside a function which is nested inside a function. This remains the same 
whether the decorator is used as a stacked decorator or not. Here is a solution: 


def trace(msg) : 
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lene aliatalenedl (iebieve)) 2 
def inner2(*args, **kwargs): 


pele > os. So msg, )) 
retval = func(*args, **kwargs) 
foieatione Ve ESS ee (iibstey,)) 


return retval 
return inner2 
wetewheim ALiMiaeie ll 


de® hoeizZomtel late ( Iimesehis) 
dek amnneiel (tune): 
def inner2(*args, **kwargs): 
jonenbione, Labora close oils 
retval = func(*args, **kwargs) 
jopeatigues all abiaveyacholig: 29 7 i185) 
return retval 
return inner2 
return innerl 


@trace('tracing funcl') 
def funcl(x, y): 
jonenumite, ses . Boas Sastre Bay 
result — Lune2 ( @<, 3) 
return result 


ChomicnZomtbell lame ( <5 a>") 
@trace('tracing func2') 
dem fune2 (content) 
print “content:", content 
return content * 3 


def test(): 
Besulee— fumed aia) “blo! ) 
print “result:", result 


test () 


3.8.1.3 More help with decorators 
There is more about decorators here: 


e Python syntax and semantics -- 
http://en.wikipedia.org/wiki/Python_syntax_and_semantics#Decorators at 
Wikipedia. 

e PythonDecoratorLibrary -- http://wiki.python.org/moin/PythonDecoratorLibrary 
at the Python Wiki has lots of sample code. 

e PEP 318 -- Decorators for Functions and Methods -- 
http://www.python.org/dev/peps/pep-03 18/ is the formal proposal and 
specification for Python decorators. 
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3.8.2 lIterables 


3.8.2.1 A few preliminaries on Iterables 
Definition: iterable (adjective) -- that which can be iterated over. 


A good test of whether something is iterable is whether it can be used in a for: 
statement. For example, if we can write for item in X:, then X is iterable. Here is 
another simple test: 


def isiterable (x): 
iy. 
y = iter (x) 
excepe Dlypeckanron,. cx: 
return False 
return True 


Some kinds of iterables: 


e Containers -- We can iterate over lists, tuples, dictionaries, sets, strings, and other 
containers. 

e Some built-in (non-container) types -- Examples: 

o A text file open in read mode is iterable: it iterates over the lines in the file. 

o The xrange type -- See XRange Type 
http://docs.python.org/lib/typesseq-xrange.html. It's useful when you want a 
large sequence of integers to iterate over. 

e Instances of classes that obey the iterator protocol. For a description of the iterator 
protocol, see Iterator Types -- http://docs.python.org/lib/typeiter.html. Hint: Type 
dir (obj) and look for "__iter__" and "next". 

e Generators -- An object returned by any function or method that contains yield. 

Exercises: 


1. Implement a class whose instances are interable. The constructor takes a list of 
URLs as its argument. An instance of this class, when iterated over, generates the 
content of the Web page at that address. 

Solutions: 


1. We implement a class that has__ iter __() and next () methods: 


alqexonere, ire 1l Nats) 


class WebPages (object): 


Que sige sella, inhells)) 9 
self.urls = urls 
self.current_index = 0 

oleae akeSiem  (syeulae se 
self.current_index = 0 


return self 
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def next (self): 

ihe (Scie Cuntent index >— Wen (sel siicils) 
raise StopIteration 

Pie = Sreulie Sie) [sveulir sielbliciaetoie = alinroles<]| 

self.current_index += 1 

B= cedar nate lhowem (aie 1) 

content = f.read() 

f.close() 

Return Content 


def test (): 
uiels == ji 
"http://www.python.org', 
Wines // ela a wal ea olevolibel orouctey// ) 


"http://en.wikipedia.org/wiki/Python_(programming_langu 


age)', 
] 
pages = WebPages (urls) 
for page in pages: 
print ‘length: %d' % (len(page), ) 
pages. =| WebPages (ue lis) 
ican VS ease o0) 
page = pages.next () 
pent Tiengehs od! =o (len(page) 7) 
page = pages.next () 
print ‘length: %d' % (len(page), ) 
page = pages.next () 
print ‘length: %d' % (len(page), ) 
page = pages.next () 
print ‘length: %d' % (len(page), ) 
wesc (() 


3.8.2.2 More help with iterables 


The itertools module in the Python standard library has helpers for iterators: 
http://docs.python.org/library/itertools.html#module-itertools 


3.9 Applications and Recipes 


3.9.1 XML -- SAX, minidom, ElementTree, Lxml 


Exercises: 


1. SAX -- Parse an XML document with SAX, then show some information (tag, 
attributes, character data) for each element. 

2. Minidom -- Parse an XML document with minidom, then walk the DOM tree 
and show some information (tag, attributes, character data) for each element. 
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Here is a sample XML document that you can use for input: 


<?xml version="1.0"?> 
<people> 
<person id="1" value="abcd" ratio="3.2"> 
<name>Alberta</name> 
<interest>gardening</interest> 
<interest>reading</interest> 
<category>5</category> 
</person> 
<person id="2"> 
<name>Bernardo</name> 
<interest>programming</interest> 
<category></category> 
<agent> 
<firstname>Darren</firstname> 
<lastname>Diddly</lastname> 
</agent> 
</person> 
<person id="3" value="efgh"> 
<name>Charlie</name> 
<interest>people</interest> 
<interest>cats</interest> 
<interest>dogs</interest> 
<category>8</category> 
<promoter> 
<firstname>David</firstname> 
<lastname>Donaldson</lastname> 
—<itenit> 
<fullname>Arnold Applebee</fullname> 
<refid>10001</refid> 
</ client 
</promoter> 
<promoter> 
<firstname>Edward</firstname> 
<lastname>Eddleberry</lastname> 
<client> 
<fullname>Arnold Applebee</fullname> 
<refid>10001</refid> 
<felieiaic> 
</promoter> 
</person> 
</people> 


ElementTree -- Parse an XML document with ElementTree, then walk the DOM 
tree and show some information (tag, attributes, character data) for each element. 
lxml -- Parse an XML document with lxml, then walk the DOM tree and show 
some information (tag, attributes, character data) for each element. 

Modify document with ElementTree -- Use ElementTree to read a document, then 
modify the tree. Show the contents of the tree, and then write out the modified 
document. 

XPath -- Ixml supports XPath. Use the XPath support in lxml to address each of 
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the following in the above XML instance document: 
o The text in all the name elements 
o The values of all the id attributes 


Solutions: 


1. We can use the SAX support in the Python standard library: 


!/usr/bin/env python 


Parse and XML with SAX. Display info about each 
element. 


Usage: 
python test_sax.py infilename 
Eo<ampike's|: 
python test_sax.py people.xml 


import sys 
from xml.sax import make_parser, handler 


class TestHandler (handler.ContentHandler): 
lene imac SsiSlae )) 2 
self.level = 0 


def show_with_level (self, value): 


fo) 


def startDocument (self): 
self.show_with_level ('Document start") 
self.level += 1 


def endDocument (self): 
self.level -= 1 
self.show_with_level ('Document end") 


def startElement (self, name, attrs): 


print "ss%s" 3 (' ' * self.level, value, 


self.show_with_level ('start element nam 


Vos © se (name, ))) 
self.level += 1 


def endElement (self, name): 


self.level -= 1 
self.show_with_level ('end element nam 
"Ss"" S$ (name, )) 


def characters(self, content): 
content = content.strip() 
it content: 


sellic  Sinowe wabiela Meal ( Velaciesicieeies2 Wes lhy 


(content, )) 


) 
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def 


def 


def test (infilename): 


parser = make_parser () 
handler = TestHandler () 
parser.setContentHandler (handler) 
parser.parse (infilename) 
usage (): 
joucalione, = “ole 
sys.exit (1) 
juavelalsay(()) <6 

Guess Byte yeliatenve || ILS || 

if den(args) l=) i: 

usage () 

infilename = args[0] 
test (infilename) 

name —— main! 
main () 


2. The minidom module contains a parse () function that enables us to read an 
XML document and create a DOM tree: 


(attr.name, 


!/usr/bin/env python 
"""Bbrocess an XML document with minidom. 


Show the document tree. 


acer. value; ») 
if (len(node.childNodes) == 1 and 
node.childNodes[0].nodeType = 


Visage: 
python minidom_walk.py [options] infilename 
wow 
AMNOee Sys 
from xml.dom import minidom 
lene evo jeicerS (core) 3 
rooe = doc. document Element 
show_node (root, 0) 
def show_node(node, level): 
count = 0 
if node.nodeType == minidom.Node.ELEMENT_NODE: 
show_level (level 
print 'tag: %s' % (node.nodeName, ) 
for key in node.attributes.keys(): 
attr = node.attributes.get (key) 
show_level (level + 1) 
print '- attribute name: %s value: "%s 
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MiinaiGdomeaNode. ME xh aN@DE) 
show_level (level + 1) 
jonenigue, 9 Clare a) Wires as 

(node.childNodes[0].data, ) 

for Chaild am mode..chilkdNodes: 
count +=" il 
show_node(child, level + 1) 

IS eben Cxoublae 


def show_level (level 
for x in range(level): 


jouealim@ie  U er 


def test(): 
args = syswargqv iil: | 
if len(args) != 1: 
pring —sdocr 
Sis ese iste (a) 
docname = args[0] 
doc = minidom.parse (docname) 
show_tree (doc) 


alae name == "main": 
import pdb; pdb.set_trace() 
test () 


3. ElementTree enables us to parse an XML document and create a DOM tree: 


!/usr/bin/env python 


"""Brocess an XML document with elementtree. 
Show the document tree. 


Usage: 
python elementtree_walk.py [options] infilename 


import sys 
from xml.etree import ElementTr als: Siz ie 


def show_tree(doc): 
root = doc.getroot () 
show_node (root, 0) 


def show_node(node, level): 
show_level (level) 
print “tag? 2s” 3s (modettag, } 
for key, value in node.attrib.iteritems(): 
show_level (level + 1) 
Deine YS ee ribube —— names os values sos. | 
(key, value, 
if node.text: 
text = node.text.strip() 
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show_level (level + 1) 

pring “= ‘text: “Ss"" = (mode.text, ) 
it node. taal: 

anil — nedes taney strain) 

show_level (level + 1) 

jonenioie US jeelakile Uh esis) 
for child in node.getchildren(): 


show mode (chailla, ieweiy 4 1) 
def show_level (level): 
for x in range(level): 
jolmesimic: t Ue 
Clee cesic () 3 
args — syswargv iil: | 
if len(args) ‘lee 
prine —sdoecr— 
sys.exit (1) 
docname = args[0] 
doc = etree.pars 
show_tree (doc) 


{i 


(docname) 


ale 


name main__': 
import pdb; pdb.set_trace() 


test () 


4. lIxml enables us to parse an XML document and create a DOM tree. In fact, since 
Ixml attempts to mimic the ElementTree API, our code is very similar to that in 
the solution to the ElementTree exercise: 


!/usr/bin/env python 
"""Brocess an XML document with elementtree. 
Show the document tree. 


Usage: 


python lxml_walk.py [options] infilename 


imports si: 
import sys 
from lxml import etree 


def show_tree (doc): 
root = doc.getroot () 


show_node (root, 


def show_node (node, 


0) 


level): 


show_level (1 
joneabone  Vineyers 


level) 


° 


Ss" % (node.tag, ) 


1c@ue ey, well 


Me ane node abkicirtilo. Hecrmiiems (): 
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show_level (level + 1) 
print '- attribute -- name: %s value: "%s 
(key, value, 
if node.text: 
text = node.text.strip() 


show_level (level + 1) 

pring “= text: “ss"" < (mode. text, ) 
if node.tail: 

Ean — node tail stig) 

show_level (level + 1) 

jonenioe Ue Seciat Mee Ue Se (iersiatl 
for child in node.getchildren(): 


show _nodei(chaillad, illeweil + 11) 


def show_level (level): 
for x in range(level): 
pring * a 


Clee eesic () 3 

args = sys.argv[1:] 

if len(args) != 1: 
jQuealine Clove 
sys.exit (1) 

docname = args[0] 

doc = etree.parse (docname) 

show_tree (doc) 


ade name == ' main": 
import pdb; pdb.set_trace() 
test () 


modify the DOM tree and write it out to a new file: 


!/usr/bin/env python 

"""Brocess an XML document with elementtree. 
Show the document tree. 

Modify the document tree and then show it again. 


Write the modified XML tree to a new file. 


Usage: 
python elementtree_walk.py [options] infilename 


outfilename 
Options: 

—hy, =—he lip Display this help message. 
Example: 

python elementtree_walk.py myxmldoc.xml 


myotherxmldoc.xml 


import sys 
import os 
import getopt 
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import time 


Use lenis 1 te 

from xml.etree import ElementTr 2s S1bie 
Om UNcommenie seo Use isan. 

from 1lxml import etree 


def show_tree(doc): 
root = doc.getroot () 
show_node (root, 0) 


def show_node(node, level): 
sino lhewe IL ihewell), 
RDMane Veage ss. cs (modentag,..)) 
for key, value in node.attrib.iteritems(): 
show_level (level + 1) 
prink "= ateribute —= names ss. values “ss 93 
(key, value, ) 
if node.text: 
text = node.text.strip() 


show_level (level + 1) 

peine “=atexts "Se" 3 Wnodestext, ) 
iit MOGeS et aclu 

tail = node.tail.strip() 

show_level (level + 1) 

jonenitae MS ic ebibe Ue eject ile) 
for child in node.getchildren(): 


Slaven aver e(elanl Ikely kes a 1) 


def show_level (level): 
for x in range(level): 
jQueauione v ae 


def modify_tree(doc, tag, attrname, attrvalue): 
root = doc.getroot () 
modify_node(root, tag, attrname, attrvalue) 


def modify_node (node, tag, attrname, attrvalue): 
if node.tag == tag: 
node.attrib[attrname] = attrvalue 
for child in node.getchildren(): 
modify_node(child, tag, attrname, attrvalue) 


def test (indocname, outdocname) : 


doc = etree.parse (indocname) 
show_tree (doc) 

je ate ied SO) 

date = time.ctime() 


modify_tree(doc, 'person', 'date', date) 
show_tree (doc) 


write_output = False 
if os.path.exists (outdocname) : 
response = raw_input('Output file (%s) exists. 


Page 247 


A Python Book 


Over-write? (y/n): ' % 
outdocname) 
if response == 'y 
write_output = True 


T ae | 
else: 
write_output = True 
if write_output: 
doc.write (outdocname) 
print ‘Wrote modified XML tree to %s' % 
outdocname 
else: 
print ‘Did not write output file.’ 


def usage(): 
jouealine Clore 
sys.exit (1) 


leu iimelalian(( 2 
args = sys.argv[1:] 
eens 
Opts, args = Ggetopt.getope (args, “h', |i*help', 
I) 
CxCepie: 
usage () 
One joe, Weill” alia\ oyonesss 
ilety, 2@ pole weakTaye (Gv bee ical ts) 
usage () 
if len(args) != 
usage () 
indocname = args[0] 
Cuedocname = args | 1] 
test (indocname, outdocname) 


eae 


aL ie name == ' | main__': 
import pdb; pdb.set_trace() 
main () 


Notes: 

o The above solution contains an import statement for ElementTree and 
another for xml. The one for xml is commented out, but you could change 
that if you wish to use lxml instead of ElementTree. This solution will work 
the same way with either ElementTree or lxml. 

6. When we parse and XML document with Ixml, each element (node) has an 
xpath () method. 


test_xpath.py 


from lxml import etree 


close esis ()) 3 
eleie) SS Seis OclesiS (( /oleColleS seal Y) 
root = doc.getroot () 
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peink Look expat h(t mame/ text () 
joneainie, idevolc. joreneian((ly/ /(Claielll!)) 


test () 


And, when we run the above code, here is what we see: 


S python test_xpath.py 
[ VAilibeinka, “Berinacdo!, “Charlie” | 
[tales Ae een a 


For more on XPath see: XML Path Language (XPath) -- 
http://www.w3.org/TR/xpath 


3.9.2 Relational database access 


You can find information about database programming in Python here: Database 
Programming -- http://wiki.python.org/moin/DatabaseProgramming/. 


For database access we use the Python Database API. You can find information about it 
here: Python Database API Specification v2.0 -- 
http://www.python.org/dev/peps/pep-0249/. 


To use the database API we do the following: 


1. Use the database interface module to create a connection object. 

Use the connection object to create a cursor object. 

Use the cursor object to execute an SQL query. 

Retrieve rows from the cursor object, if needed. 

Optionally, commit results to the database. 

Close the connection object. 

Our examples use the gadfly database, which is written in Python. If you want to use 
gadfly, you can find it here: http://gadfly.sourceforge.net/. gadfly is a reasonable 
choice if you want an easy to use database on your local machine. 


AnNRWN 


Another reasonable choice for a local database is sqlite3, which is in the Python 
standard library. Here is a descriptive quote from the SQLite Web site: 


"SQLite is a software library that implements a self-contained, 
serverless, zero-configuration, transactional SQL database engine. 
SQLite is the most widely deployed SQL database engine in the world. 
The source code for SQLite is in the public domain." 


You can learn about it here: 


e sqlite3 - DB-API 2.0 interface for SQLite databases -- 
http://docs.python.org/library/sqlite3.html 
e SQLite home page -- http://www.sqlite.org/ 
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e The pysqlite web page -- http://oss.itsystementwicklung.de/trac/pysqlite/ 
If you want or need to use another, enterprise class database, for example PostgreSQL, 
MySQL, Oracle, etc., you will need an interface module for your specific database. You 
can find information about database interface modules here: Database interfaces -- 
http://wiki.python.org/moin/DatabaseInterfaces 


Excercises: 


1. 
2 


4. 


Write a script that retrieves all the rows in a table and prints each row. 

Write a script that retrieves all the rows in a table, then uses the cursor as an 
iterator to print each row. 

Write a script that uses the cursor's description attribute to print out the name 
and value of each field in each row. 

Write a script that performs several of the above tasks, but uses sql ite3 instead 
of gadfly. 


Solutions: 


1. 


Ze 


We can execute a SQL query and then retrieve all the rows with 
ireicClna lil () ' 


Import. Gadi ly 


def test (): 

connection = gadfly.connect ("dbtest1", 
Moulkciaites Gio anigs)) 

cur = connection.cursor() 

Cullie Sxecuies ('Sellece — wictoim jollewicscls Cuwceie ley 
p_name') 

rows = cur.fetchall() 

for row in rows: 

joueanone WAS ees Vp Teron 
connection.close() 


test () 


The cursor itself is an iterator. It iterates over the rows returned by a query. So, 
we execute a SQL query and then we use the cursor in a for: statement: 


LMP ewie Gace ly, 


Glese ieeisie ()) 2 

connection = gadfly.connect ("dbtest1", 
“olantsdbdinr”) 

cur = connection.cursor () 

Culie Erecwies ('selkece ~~ wicoim jolleimicscls Gucleie ley 
p_name') 


FO seow am) Gui 
print row 
connection.close() 
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test () 


3. The description attribute in the cursor is a container that has an item describing 
each field: 


import gadfly 


def test (): 
Cule Execs (i Selece ~ wisi jolleimcsclo Ciucleie ly 
p_name') 
EOE Ele an Cum description: 
joueakinves Wiealienlyera Ay sralkeviko! 
rows = cur.fetchall() 
for row in rows: 
for idx, field in enumerate (row): 


@omesing = Yaseg Mogi) os 
(Ciba, clesiciealjore alto [| akeb< || [Ol seat), )) 
jovealions exeaiceinie p 
joneabiahe: 


connection.close() 


eeSic ()) 
Notes: 
o The comma at the end of the print statement tells Python not to print a 
new-line. 


o The cur.description is a sequence containing an item for each field. 
After the query, we can extract a description of each field. 
4. The solutions using sqlite3 are very similar to those using gadfly. For 
information on sqlite3, see: sqlite3 — DB-API 2.0 interface for SQLite 
databases http://docs.python.org/library/sqlite3.html#module-sqlite3. 


#!/usr/bin/env python 


Perform operations on sqlite3 (plants) database. 


Usage: 

pyeEhon pyodbuapipy command lacgl,. 324. | 
Commands: 

Creat Greate new database. 

show —— show Convenes, OF database. 

add -- add row to database. Requires 3 args (name, 
descrip, rating). 

cle lee remove row from database. Requires 1 arg 
(name) . 
Examples: 


python testl.py create 

Ppyehon restl py show 

python testl.py add crenshaw "The most succulent 
melon" 10 

python testl.py delete lemon 
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import sys 
import sqlite3 


Values = [ 
(o Kemonts;, ““oright and velilow" - ly!) 
(“peach!', Ysucceullent ) 7 9") 7 
('banana', 'smooth and creamy', '8') 
(nectarcineY, Ytangy and tasty!, "9 
('orange', 'sweet and tangy', '8'), 
] 


Field_defs = [ 
"p_name varchar', 
“pL descrip varchar, 
"p_rating integer', 
Wey leslie abinle; weleielayelie YF 


def createdb(): 


connection = sqlite3.connect ('sqlite3plantsdb') 
cursor = connection.cursor () 
ql = "create table plantsdb (%s)" % (', 


'. join (Field_defs) ) 
pmeineg “ereate qi: 2s) 3 ql 
GUliesOue, ExSCwheS ((e1)) 


ql = "create index indexl on plantsdb(p_name)" 
CUBSOm sexe eules (Gil) 
ql = "insert into plantsdb (p_name, p_descrip, 


( 
perating) wales i! ss), fos) cos)? 

for spec in Values: 

q2 = ql % spec 

joucaliahe Wiopeeg Wrath or (2, 

cursor.execute (q2) 
connection.commit () 
showdb1 (cursor) 


connection.close() 


def showdb(): 
connection, cursor = opendb() 
showdb1 (cursor) 
connection.close() 


def showdbl1 (cursor): 


CuiesoOw excuse (selece ~ iicom Olleinesclo Ciclo lov 
jO_ salen) 

hr () 

description = cursor.description 


Page 252 


A Python Book 


pLIne deseripe ion 
PLink “deseripeion: 
Om LOwdescripeilon anidesciript von: 


joueniaie  Y $s' % (rowdescription, ) 
en@) 
rows = cursor.fetchall () 


print rows 
jonealione, Vaetoyiasyg ¥ 
for row in rows: 
joueakione 3S oF (GOW, ) 
Joe X()) 
prink “content: ! 
for Fow an rows: 


descrip = row[1] 

name = row[0] 

rating = '%s' % row[2] 
joueaimie | SS6SCS! Sf 


On] 
~ 


name.ljust (1 
rating.rjust(4), ) 


desermip-. aust (30), 


def addtodb(name, descrip, rating): 
IENENAS 
rating = int (rating) 
except ValueError, exp: 
print ‘Error: rating must be integer.' 


TSN Bhi ig 
connection, cursor = opendb() 
Guliesiouwe -execuics ("Selec ~ scicoml Sleimesiclo yes jones 
= 'Ss'" %& name) 
rows = cursor.fetchall() 
if len(rows) > 0: 
ql = "update plantsdb set p_descrip='%s"', 
p_rating='%Ss' where p_name='%s'" 3% ( 


descrip, rating, name, ) 
joucalinye Vicia cu 
cursor.execute (ql) 
connection.commit () 
print 'Updated' 
else: 
cursor.execute ("insert into plantsdb values 
Gcsl VS! met) & ( 
name, descrip, rating) ) 
connection.commit () 
print 'Added' 
showdb1 (cursor) 
Connection. close) 


def deletefromdb (name): 


connection, cursor = opendb() 
@iblesioue ,erSebhes (/isielliereie, -« adicreyn olleisie sicloy ‘WialSicigs Jo iach 
= 'Ss'" S$ name) 
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rows = cursor.fetchall() 
if len(rows) > 0: 
CulesOue  exxecures (Mclellere ieeom Sllemescls wiseies 
p_name='%s'" % name) 


connection.commit () 

print ‘Plant (%s) deleted.' % name 
else: 

print 'Plant (%s) does not exist.' % name 
showdb1 (cursor) 
connection.close() 


def opendb(): 
connection = sqlite3.connect ("Ssqlite3plantsdb") 
Cunsor —  Connecte lon. curso () 


return connection, cursor 


lene inie (()) 8 
joneatigie UN a “Gi() 


def usage(): 
prine Sedocr= 
sys.exit (1) 


def main(): 


euiecis) 1 Sayisty clistenie || Ile || 
if len(args) < 1: 
usage () 
cmd = args[0] 
if cmd == 'create!: 
if len(args) != 1: 
usage () 
createdb () 
elif cmd == 'show!': 
if len(args) != 1: 
usage () 
showdb () 
elif cmd == '‘add': 
if len(args) < 4: 
usage () 
name = args[1] 


descrip = args[2] 

rating = args[3] 

addtodb(name, descrip, rating) 
elif cmd == 'delete': 

if len(args) < 2: 

usage () 

name = args[1] 

deletefromdb (name) 
else: 
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3.9.3 CSV -- comma separated value files 


There is support for parsing and generating CSV files in the Python standard library. See: 
csv — CSV File Reading and Writing 
http://docs.python.org/library/csv.html#module-csv. 


Exercises: 


1. Read a CSV file and print the fields in columns. Here is a sample file to use as 


input: 
nam deseciption rating 
Lemon, Bright yellow and tart,5 
Eggplant,Purple and shiny, 6 
Tangerine, Succulent, 8 
Solutions: 


1. Use the CSV module in the Python standard library to read a CSV file: 


Wo 


Read a CSV file and print the contents in columns. 


LM Onsite eS, 


def test (infilename): 


infile = open(infilename) 

reader = csv.reader (infile) 

print '==== —————_—__-—— 
a 7 

print 'Name Description 
Rating * 

print '==== ——_-____—-—— 


PO aes! ane ieeacets 


if len(fields) == 3: 
line = '%Ss %s %s' % (fields[0].1ljust (20), 


sealenlye Feil || Salegbysie (C240) 
fields[2].1ljust (4)) 
Deine daiae 
infile.close() 


def main(): 
infilename = 'csv_report.csv' 
test (infilename) 
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aL ie name == ' main __': 


main () 


And, when run, here is what it displays: 


Name Description 

Rating 

Lemon Bright yellow and tart 
©) 

Eggplant Purple and shiny 

6 

Tangerine Sblelerbullsiaye. 

8 


3.9.4 YAML and PyYAML 


YAML is a structured text data representation format. It uses indentation to indicate 
nesting. Here is a description from the YAML Web site: 


"YAML: YAML Ain't Markup Language 
"What It Is: YAML is a human friendly data serialization standard for 
all programming languages." 

You can learn more about YAML and PyY AML here: 


e The Official YAML Web Site -- http://yaml.org/ 
e PyYAML.org - the home of various YAML implementations for Python -- 
http://pyyaml.org/ 
e The YAML 1.2 specification -- http://yaml.org/spec/1.2/ 
Exercises: 


1. Read the following sample YAML document. Print out the information in it: 


american: 

- Boston Red Sox 

- Detroit Tigers 

— New York Yankees 
national: 

— New York Mets 

= (Cmalecice bios 

- Atlanta Braves 


2. Load the YAML data used in the previous exercise, then make a modification (for 
example, add "San Francisco Giants" to the National League), then dump the 
modified data to a new file. 

Solutions: 
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1. Printing out information from YAML is as "simple" as printing out a Python data 
structure. In this solution, we use the pretty printer from the Python standard 


library: 


import yaml 
import pprint 


def test(): 
infile = open('testl.yaml') 
data = yaml.load(infile) 
infile.close() 
pprint.pprint (data) 


test () 


We could, alternatively, read in and then "load" from a string: 


import yaml 
IMO jSjoueauiane 


def test(): 
infile = open('testl.yaml') 
data_str = infile.read() 


infile.close() 
data = yaml.load(data_str) 
jojgieabinne s|Sjenaniaye ((Cleyers!), 


test () 


2. The YAML dump () function enables us to dump data to a file: 


Notes: 


import yaml 
import pprint 


def test (): 
infile = open('testl.yaml', 'r') 
data = yaml.load(infile) 
infile.close() 
data['national'].append('San Francisco Giants") 
outfile = open('testl_new.yaml', 'w') 


yaml.dump (data, outfile) 
outfile.close() 


test () 


o If we want to produce the standard YAML "block" style rather than the "flow" 
format, then we could use: 


yaml.dump (data, outfile, default_flow_style=False) 
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3.9.5 Json 


Here is a quote from Wikipedia entry for Json: 


"JSON (pronounced 'Jason’), short for JavaScript Object Notation, is a 
lightweight computer data interchange format. It is a text-based, 
human-readable format for representing simple data structures and 
associative arrays (called objects)." 


The Json text representation looks very similar to Python literal representation of Python 
builtin data types (for example, lists, dictionaries, numbers, and strings). 


Learn more about Json and Python support for Json here: 


e Introducing JSON -- http://json.org/ 
e Json at Wikipedia -- http://en.wikipedia.org/wiki/Json 
e python-json -- http://pypi.python.org/pypi/python-json 
e simplejson -- http://pypi.python.org/pypi/simplejson 
Excercises: 
1. Write a Python script, using your favorite Python Json implementation (for 


example python—json or simplejson), that dumps the following data 
structure to a file: 


Data = { 
VEOCk and, rolls 
['Elis', 'The Beatles', 'The Rolling Stones',], 
UG Ouintetayave 
[ WElitvte Neilson") "Hank Wallaiiamc 9] 


} 


2. Write a Python script that reads Json data from a file and loads it into Python data 
structures. 
Solutions: 


1. This solution uses simplejson to store a Python data structure encoded as Json 
in a file: 


import simplejson as json 


Data = { 
MEOck ame: teoOuule: 
['Elis', 'The Beatles', 'The Rolling Stones',], 


VeouniEny =: 
['Willie Nelson', 'Hank Williams', ] 
} 
Glee jesse () 8 
fout = open('tmpdata.json', '‘'w') 
content = json.dumps (Data) 


fout.write (content) 
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FOU. wiete ec) wm ) 
fout.close() 


test () 


2. We-can read the file into a string, then decode it from Json: 


import simplejson as json 


def test(): 
fin — open tmpdaitea. Son.) \ 2.) 
content = fin.read() 
fin.close() 
data = json.loads (content) 
joueiinie, Cees) 


test () 


Note that you may want some control over indentation, character encoding, etc. For 
simplejson, you can learn about that here: simplejson - JSON encoder and decoder -- 
http://simplejson. googlecode.com/svn/tags/simplejson-2.0.1/docs/index.html. 
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4 Part 4 -- Generating Python Bindings for XML 


This section discusses a specific Python tool, specifically a Python code generator that 
generates Python bindings for XML files. 


Thus, this section will help you in the following ways: 


1. It will help you learn to use a specific tool, namely generateDS. py, that 
generates Python code to be used to process XML instance documents of a 
particular document type. 

2. It will help you gain more experience with reading, modifying and using Python 
code. 


4.1 Introduction 


Additional information: 


e If you plan to work through this tutorial, you may find it helpful to look at the 
sample code that accompanies this tutorial. You can find it in the distribution 
under: 


ie WUeonaalcshlly/ 
tutorial/Code/ 


e You can find additional information about generateDS.py here: 
http://http://www.davekuhlman.org/#generateds-py 


That documentation is also included in the distribution. 
generateDS.py generates Python data structures (for example, class definitions) from 
an XML schema document. These data structures represent the elements in an XML 
document described by the XML schema. generateDS. py also generates parsers that 
load an XML document into those data structures. In addition, a separate file containing 
subclasses (stubs) is optionally generated. The user can add methods to the subclasses in 
order to process the contents of an XML document. 


The generated Python code contains: 


e Acclass definition for each element defined in the XML schema document. 

e A main and driver function that can be used to test the generated code. 

e A parser that will read an XML document which satisfies the XML schema from 
which the parser was generated. The parser creates and populates a tree structure 
of instances of the generated Python classes. 

e Methods in each class to export the instance back out to XML (method export) 
and to export the instance to a literal representing the Python data structure 
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(method exportLiteral). 
Each generated class contains the following: 


e A constructor method (__init__), with member variable initializers. 

e Methods with names get_xyz and set_xyz for each member variable "xyz" 
or, if the member variable is defined with maxOccurs="unbounded", 
methods with names get_xyz, set_xyz, add_xyz, and insert_xyz. 
(Note: If you use the --use-old-getter-setter, then you will get 
methods with names like get Xyz and set Xyz.) 

e A build method that can be used to populate an instance of the class from a 
node in an ElementTree or Lxml tree. 

e Anexport method that will write the instance (and any nested sub-instances) to 
a file object as XML text. 

e AnexportLiteral method that will write the instance (and any nested 
sub-instances) to a file object as Python literals (text). 

The generated subclass file contains one (sub-)class definition for each data 
representation class. If the subclass file is used, then the parser creates instances of the 
subclasses (instead of creating instances of the superclasses). This enables the user to 
extend the subclasses with "tree walk" methods, for example, that process the contents of 
the XML file. The user can also generate and extend multiple subclass files which use a 
single, common superclass file, thus implementing a number of different processes on the 
same XML document type. 


This document introduces the user to generateDS.py and walks the user through 
several examples that show how to generate Python code and how to use that generated 
code. 


4.2 Generating the code 
Note: The sample files used below are under the tutorial/Code/ directory. 


Use the following to get help: 


S$ generateDS.py --help 


I'll assume that generateDS. py is in a directory on your path. If not, you should do 
whatever is necessary to make it accessible and executable. 


Here is a simple XML schema document: 


And, here is how you might generate classes and subclasses that provide data bindings (a 
Python API) for the definitions in that schema: 


S generateDS.py -o people_api.py -s people_sub.py people.xsd 
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And, if you want to automatically over-write the generated Python files, use the —f 
command line flag to force over-write without asking: 


S generateDS.py -f -o people_api.py -s people_sub.py people.xsd 


And, to hard-wire the subclass file so that it imports the API module, use the --super 
command line file. Example: 


S generateDS.py -o people_api.py people.xsd 
S generateDS.py -s people_appll.py --super=people_api people.xsd 


Or, do both at the same time with the following: 


S generateDS.py -o people_api.py -s people_appll.py 
super=people_api people.xsd 


And, for your second application: 


S$ generateDS.py -s people_appl2.py --super=people_api people.xsd 


If you take a look inside these two "application" files, you will see and import statement 
like the following: 


import ??? as supermod 


If you had not used the --super command line option when generating the 
"application" files, then you could modify that statement yourself. The --super 
command line option does this for you. 


You can also use the The graphical front-end to configure options and save them in a 
session file, then use that session file with generateDS.py to specify your command 
line options. For example: 


S generateDS.py —-session=test01.session 


You can test the generated code by running it. Try something like the following: 


S python people_api.py people.xml 


Or: 


S python people_appll.py people.xml 


Why does this work? Why can we run the generated code as a Python script? -- If you 
look at the generated code, down near the end of the file you'll find a main () function 
that calls a function named parse (). The parse function does the following: 


1. Parses your XML instance document. 
2. Uses your generated API to build a tree of instances of the generated classes. 
3. Uses the export () methods in that tree of instances to print out (export) XML 
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that represents your generated tree of instances. 
Except for some indentation (ignorable whitespace), this exported XML should be the 
same as the original XML document. So, that gives you a reasonably thorough test of 
your generated code. 


And, the code in that parse () function gives you a hint of how you might build your 
own application-specific code that uses the generated API (those generated Python 
classes). 


4.3 Using the generated code to parse and export an XML document 


Now that you have generated code for your data model, you can test it by running it as an 
application. Suppose that you have an XML instance document peoplel.xm1 that 
satisfies your schema. Then you can parse that instance document and export it (print it 
out) with something like the following: 


S python people_api.py peoplel.xml 


And, if you have used the -~super command line option, as I have above, to connect 
your subclass file with the superclass (API) file, then you could use the following to do 
the same thing: 


S python people_appll.py peoplel.xml 


4.4 Some command line options you might want to know 


You may want to merely skim this section for now, then later refer back to it when some 
of these options are are used later in this tutorial. Also, remember that you can get 
information about more command line options used by generateDS. py by typing: 


S$ python generateDS.py —-—help 


and by reading the document at http://www.davekuhlman.org/#generateds-py 


0 
Generate the superclass module. This is the module that contains the implementation 
of each class for each element type. So, you can think of this as the implementation of 
the "data bindings" or the API for XML documents of the type defined by your XML 
schema. 

s 


Generate the subclass module. You might or might not need these. If you intend to 
write some application-specific code, you might want to consider starting with these 
skeleton classes and add your application code there. 
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super 


This option inserts the name of the superclass module into an import statement in 
the subclass file (generated with "-s"). If you know the name of the superclass file in 
advance, you can use this option to enable the subclass file to import the superclass 
module automatically. If you do not use this option, you will need to edit the subclass 
module with your text editor and modify the import statement near the top. 


root-element='"'element-name" 


Use this option to tell generateDS.py which of the elements defined in your XM 
schema is the "root" element. The root element is the outer-most (top-level) element 
in XML instance documents defined by this schema. In effect, this tells your 
generated modules which element to use as the root element when parsing and 
exporting documents. 


generateDS.py attempts to guess the root element, usually the first element 
defined in your XML schema. Use this option when that default is not what you want. 


member-specs=listidict 


Suppose you want to write some code that can be generically applied to elements of 
different kinds (element types implemented by several different generated classes. If 
so, it might be helpful to have a list or dictionary specifying information about each 
member data item in each class. This option does that by generating a list or a 
dictionary (with the member data item name as key) in each generated class. Take a 
look at the generated code to learn about it. In particular, look at the generated list or 
dictionary in a class for any element type and also at the definition of the class 
_MemberSpec generated near the top of the API module. 


version 


Ask generateDS.py to tell you what version it is. This is helpful when you want 
to ask about a problem, for example at the generateds-users email list 
(https://lists.sourceforge.net/lists/listinfo/generateds-users), and want to specify which 
version you are using. 


4.5 The graphical front-end 


There is also a point-and-click way to run generateDsS. It enables you to specify the 
options needed by generateDS. py through a graphical interface, then to run 
generateDS.py with those options. It also 


You can run it, if you have installed generateDS, by typing the following at a 
command line: 
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S generateds_gui.py 


After configuring options, you can save those options in a "session" file, which can be 
loaded later. Look under the File menu for save and load commands and also consider 
using the "--session" command line option. 


Also note that generateDS. py itself supports a "--session" command line option that 
enables you to run generateDS.py with the options that you specified and saved with 
the graphical front-end. 


4.6 Adding application-specific behavior 


generateDS.py generates Python code which, with no modification, will parse and 
then export an XML document defined by your schema. However, you are likely to want 
to go beyond that. In many situations you will want to construct a custom application that 
processes your XML documents using the generated code. 


4.6.1 Implementing custom subclasses 


One strategy is to generate a subclass file and to add your application-specific code to 
that. Generate the subclass file with the "-s" command line flag: 


E generateDS.py -s myapp.py people.xsd 


Now add some application-specific code to myapp. py, for example, if you are using the 
included "people" sample files: 


class peopleTypeSub (supermod.people): 
def __init__(self, comments=None, person=None, programmer=None, 
python_programmer=None, java_programmer=None) : 
supermod.people.__init__(self, comments, person, programmer, 
python_programmer, 
java_programmer) 
def fancyexport (self, outfile): 
outfile.write('Starting fancy export') 
for person in self.get_person(): 
person. fancyexport (outfile) 
supermod.people.subclass = peopleTypeSub 
# end class peopleTypeSub 


class personTypeSub (supermod.person) : 
def __init__(self, vegetable=None, fruit=None, ratio=None, 
id=None, value=None, 
name=None, interest=None, category=None, agent=None, 
preometer-Nome, 
description=None): 
supermod.person.__init__ (self, vegetable, fruit, ratio, id, 


value, 
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name, interest, category, agent, promoter, description) 
def fancyexport (self, outfile): 
outfile.write('Fancy person export -- name: %s' % 
self.get_name(), ) 
supermod.person.subclass = personTypesub 
# end class personTypeSub 


4.6.2 Using the generated "API" from your application 


In this approach you might do things like the following: 


e import your generated classes. 

e Create instances of those classes. 

e Link those instances, for example put "children" inside of a parent, or add one or 
more instances to a parent that can contain a list of objects (think "maxOccurs" 
greater than | in your schema) 


Get to know the generated export API by inspecting the generated code in the superclass 
file. That's the file generated with the "-o" command line flag. 


What to look for: 


e Look at the arguments to the constructor (__init__) to learn how to initialize 
an instance. 

e Look at the "getters" and "setters" (methods name get xxx and set xxx, to learn 
how to modify member variables. 

e Look for a method named addxxx for members that are lists. These correspond 
to members defined with maxOccurs="n", where n> l. 

e Look at the build methods: build, buildChildren, and 
buildAttributes. These will give you information about how to construct 
each of the members of a given element/class. 


Now, you can import your generated API module, and use it to construct and manipulate 
objects. Here is an example using code generated with the "people" schema: 


import sys 
import people_api as api 


def test (names): 
people = api.peopleType () 
for count, name in enumerate (names) : 
el oC san (Coume a ly 9) 
person = api.personType(name=name, id=id) 
people.add_person (person) 
people.export (sys.stdout, 0) 


test (['albert', 'betsy', 'charlie']) 


Run this and you might see something like the following: 
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S python tmp.py 
<Oeople > 
<person id="1"> 
<name>albert</name> 
</person> 
<person id="2"> 
<name>bet sy</name> 
</person> 
<person id="3"> 
<name>charlie</name> 
</person> 
</people> 


4.6.3 A combined approach 


Note: You can find examples of the code in this section in these files: 


tutorial/Code/upcase_names.py 
tutorial/Code/upcase_names_appl.py 


Here are the relevant, modified subclasses (upcase_names_appl.py): 


import people_api as supermod 


class peopleTypeSub (supermod.peopleType) : 

def __init__(self, comments=None, person=None, 
specialperson=None, programmer=None, python_programmer=None, 
java_programmer=None) : 
super (peopleTypeSub, self) .__init__(comments, person, 
specialperson, programmer, python_programmer, java_programmer, ) 

def upcase_names (self): 

for person in self.get_person(): 
person.upcase_names () 

supermod.peopleType.subclass = peopleTypeSub 
# end class peopleTypeSub 


class personTypeSub (supermod.personType) : 
def __init__(self, vegetable=None, fruit=None, ratio=None, 
id=None, value=None, name=None, interest=None, category=None, 
agent=None, promoter=None, description=None, range_=None, 
extensiontype_=None) : 
super (personTypeSub, self).__init__(vegetable, fruit, ratio, 
id, value, name, interest, category, agent, promoter, description, 
range_, extensiontype_, ) 
def upcase_names (self): 
self.set_name(self.get_name() .upper()) 
supermod.personType.subclass = personTypeSub 
# end class personTypeSub 


Notes: 


e These classes were generated with the "-s" command line option. They are 
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subclasses of classes in the module people_api, which was generated with the 
"-o" command line option. 

e The only modification to the skeleton subclasses is the addition of the two 
methods named upcase_names (). 

e Inthe subclass peopleTypeSub, the method upcase_names () merely walk 
over its immediate children. 

e Inthe subclass personTypeSub, the method upcase_names () just converts 
the value of its "name" member to upper case. 

Here is the application itself (upcase_names. py): 


Meenas Sis! 
import upcase_names_appl as appl 


def create_people (names) : 
people = appl.peopleTypeSub () 
for count, name in enumerate (names) : 
ic, — ods 3 (coume + 1.) 
person = appl.personTypeSub (name=name, id=id) 
people.add_person (person) 
return people 


def main(): 
names = ['albert', 'betsy', 'charlie'] 
people = create_people (names) 
pLine, YBetrore: * 
Pcs cerqeowe (Swiss aseclouke, IL) 
people.upcase_names () 
joresbionc, es (0) 
print 'After:' 
PSCOPLe Sxowe (Svs csecouie, IL) 


main () 


Notes: 


e The create_people() function creates a peopleTypeSub instance with 
several personTypeSub instances inside it. 
And, when you run this mini-application, here is what you might see: 


S$ python upcase_names.py 
Bemomer. 
—peop ley 
<person id="1"> 
<name>albert</name> 
</person> 
<person id="2"> 
<name>bet sy</name> 
</person> 
<person id="3"> 
<name>charlie</name> 
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</person> 
</people> 
After: 
<jxsojolhe = 
<person id="1"> 
<name>ALBERT</name> 
</person> 
<person id="2"> 
<name>BETSY</name> 
</person> 
<person id="3"> 
<name>CHARLIE</name> 
</person> 
</people> 


4.7 Special situations and uses 


4.7.1 Generic, type-independent processing 


There are times when you would like to implement a function or method that can perform 
operations on a variety of members and that needs type information about each member. 


You can get help with this by generating your code with the "--member-specs" command 
line option. When you use this option, generateDS.py add a list or a dictionary 
containing an item for each member. If you want a list, then use "--member-specs=list", 
and if you want a dictionary, with member names as keys, then use 
"--member-specs=dict". 


Here is an example -- In this example, we walk the document/instance tree and convert 
all string simple types to upper case. 


Here is a schema (Code/member_specs.xsd): 


<?xml version="1.0"?> 
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> 


<xs:element name="contact-list" type="contactlistType" /> 


<xs:complexType name="contactlistType"> 

OG 5 SSeiiemela> 

<xs:element name="description" type="xs:string" /> 
“<spelememe meane="Comceee” cyjse=iCoOmiceer lyjosY 
maxOccurs="unbounded" /> 

</xs:sequence> 

<xs:iattribute name="locator" type="xs:string" /> 
</xs:complexType> 


<xs:complexType name="contactType"> 
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x5 2 Secuenes> 
<xs:element name="first-—name" type="xs:string"/> 
<xs:element name="last-name" type="xs:string"/> 
<xs:element name="interest" type="xs:string" 
maxOccurs="unbounded" /> 
<xs:element name="category" type="xs:integer"/> 
</xs:sequence> 
<xs:attribute name="id" type="xs:integer" /> 
<xs:iattribute name="priority" type="xs:float" /> 
<xs:iattribute name="color-—code" type="xs:string" /> 
</xs:complexType> 


</xs:schema> 


4.7.1.1 Step 1 -- generate the bindings 


We generate code with the following command line: 


$ generateDS.py -f \ 
-o member_specs_api.py \ 
-s member_specs_upper.py \ 
super=member_specs_api \ 
member-specs=list \ 
member_specs.xsd 


Notes: 


e We generate the member specifications as a list with the command line option 
member-—specs=1ist. 
e We generate an "application" module with the -s command line option. We'll put 
our application specific code in member_specs_upper.py. 


4.7.1.2 Step 2 -- add application-specific code 


And, here is the subclass file (member_specs_upper.py, generated with the "-s" 
command line option), to which we have added a bit of code that converts any string-type 
members to upper case. You can think of this module as a special "application" of the 
generated classes. 


#!/usr/bin/env python 


# 
# member_specs_upper.py 


# 


# 
# Generated Tue Nov 9 15:54:47 2010 by generateDS.py version 2.2a. 
# 


soos, Sis 
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import member_specs_api as supermod 


euncen, — Nome 

Verbose_import_ = False 

( XMLParser_import_none, XMLParser_import_lxml, 
XMLParser_import_elementtree 
) = range (3) 

XMLParser_import_library = None 

eaves 


# Lxml 
from 1lxml import etree as etree_ 
XMLParser_import_library = XMLParser_import_lxml 
if Verbose_import_: 
print ("running with lxml.etree") 
Cxcepe IMporehmrom: 
Bry: 
# cElementTree from Python 2.5+ 
import xml.etree.cElementTree as etree_ 
XMLParser_import_library = XMLParser_import_elementtree 
if Verbose_import_: 
print ("running with cElementTree on Python 2.5+") 
CxGepe Tnporehinre nr: 
tiny: 


# ElementTree from Python 2.5+ 
import xml.etree.ElementTr eis SieiceS _ 
XMLParser_import_library = XMLParser_import_elementtree 
if Verbose_import_: 
print ("running with ElementTree on Python 2.5+") 
exqeejoie, lMumerouee lieve + 
ery 
# normal cElementTree install 
import cElementTree as etree_ 
XMLParser_import_library = 
XMLParser_import_elementtree 
if Verbose_import_: 
print ("running with cElementTree") 
excepe Imporeknror: 
try: 
# normal ElementTree install 
import elementtree.ElementTree as etree_ 
XMLParser_import_library = 
XMLParser_import_elementtree 
if Verbose_import_: 
print ("running with ElementTree") 
evscSjoue dlulexouew|diewone + 
raise ImportError ("Failed to import ElementTree 
from any known place") 


det parsexmila(sargs,, ~““kwaregs))s: 
if (XMLParser_import_library == XMLParser_import_lxml and 
‘parser’ not in kwargs): 
# Use the lxml ElementTree compatible parser so that, e.g., 
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# we ignore comments. 

Iveco) Vi eleseue! | = ScikeS_ . mCi coiMili2ciiesisic ()) 
COC =] SESS _,SeeSe (args, —lewenee/s)) 
Keturnm doe 


ct 
# Globals 
4 


ExternalEncoding = ‘ascii' 


# 
# Utility funtions needed in each generated class. 


# 


def upper_elements (obj): 
for item in obj.member_data_items 


if item.get_data_type() == 'xs:string': 
name = remap (item.get_name() ) 
vall = getattr(obj, name) 


Ge che Shinsineiee (Neil IlskSte))e 
for idx, val2 in enumerate(vall): 
vall[idx] = val2.upper () 
else: 
setattr(obj, name, vall.upper()) 


def remap (name): 
newname = name.replace('-', '_") 
return newname 


i 
# Data representation classes 


# 


class contactlistTypeSub (supermod.contactlistType) : 
def __ init__(self, locator=None, description=None, contact=None) : 
super (contactlistTypeSub, self).__init__(locator, 
description, contact, } 
def upper(self): 
upper_elements (self) 
Oe olmalitel Glial Gyeulic pei. Tooikecroig (()) 6 
child.upper () 
supermod.contactlistType.subclass = contactlistTypeSub 
# end class contactlistTypeSub 


class contactTypeSub (supermod.contactType) : 
def __ init__(self, priority=None, color_code=None, id=None, 
first_name=None, last_name=None, interest=None, category=None): 
super (contactTypeSub, self).__init__(priority, color_code, 
id, first_name, last_name, interest, category, ) 
def upper(self): 
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upper_elements (self) 
supermod.contactType.subclass = contactTypesSub 
# end class contactTypeSub 


def get_root_tag (node): 


cals) = SWlosieioCl. Lee, PereicSieit_ -ilec lat (imvorelS ieeke))) a epeowyers )) [IL] 
imoenwlSlhelsisy = INNS 
if hasattr(supermod, tag): 

rootClass = getattr(supermod, tag) 


return tag, Kooeclass 


def parse (inFilename): 
doc = parsexml_(inFilename) 
rootNode = doc.getroot () 
rootTag, rootClass = get_root_tag (rootNode) 
if rootClass is None: 
roOuLag = “Contacte —lise" 
rootClass = supermod.contactlistType 
GOoEOba — GoorClass. factory () 
rootObj.build(rootNode) 
# Enable Python to collect the space used by the DOM. 
doc = None 
sys.stdout.write('<?xml version="1.0" ?>\n') 
rootObj.export (sys.stdout, 0, name_=rootTag, 
namespacedef_='') 
coe = None 
return rootObj 


def parseString(inString): 
irom SteinglO impoerk String l© 
doc = parsexml_(StringI0O(inString) ) 
rootNode = doc.getroot () 
rootTag, rootClass = get_root_tag (rootNode) 
Th rooucilass) usm None: 
MOCekec = Teomesioe— Iasi! 
rootClass = supermod.contactlistType 
rootObj = rootClass.factory () 
rootObj.build(rootNode) 
# Enable Python to collect the space used by the DOM. 
doc — None 
sys.stdout.write('<?xml version="1.0" ?>\n') 
rootObj.export (sys.stdout, 0, name_=rootTag, 
namespacedef_='') 
return rootObj 


def parseLiteral (inFilename) : 
doc = parsexml_(inFilename) 
rootNode = doc.getroot () 
rootTag, rootClass = get_root_tag (rootNode) 
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ww 


def 


if rootClass is None: 
rOOthag — YContace—list” 
rootClass = supermod.contactlistType 
GOCEOb Gg) — BooeC lacs. Eackony (|) 
rootObj.build(rootNode) 
# Enable Python to collect the space used by the DOM. 
doc = None 
sys.stdout.write('#from member_specs_api import *\n\n') 
sys.stdout.write('import member_specs_api as model_\n\n') 


sys.stdout.write('rootObj = model_.contact_list(\n') 
rootObj.exportLiteral(sys.stdout, 0, name_="contact_list") 
sys.stdout.write(')\n"') 


return rootObj 


USAGE_TEXT = "te" 
WsieisiSs joyyelieim 222 Gexy <ailiolr al Wsioxeiae> 


usage (): 
print USAGE_TEXT 


sys.exit (1) 


def main(): 
ards! = Sys .angm |] 
if len(args) != 1: 
usage () 
infilename = args[0] 
root = parse (infilename) 
site name == '  main__': 
#import pdb; pdb.set_trace() 
main() 
Notes: 
e We add the functions upper_elements and remap that we use in each 
generated class. 
e Notice how the function upper_elements calls the function remap only on 
those members whose type is xS: string. 
e Ineach generated (sub-)class, we add the methods that walk the DOM tree and 
apply the method (upper) that transforms each xs: string value. 
4.7.1.3 Step 3 -- write a test/driver harness 
Here is a test driver (member_specs_test.py) for our (mini-) application: 


#!/usr/bin/env python 
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# 
# member_specs_test.py 


# 


import sys 
import member_specs_api as supermod 
import member_specs_upper 


def process (inFilename) : 

doc = supermod.parsexml_(inFilename) 

rootNode = doc.getroot () 

rootClass = member_specs_upper.contactlistTypeSub 

rOOtOb] = rootClasis. factory () 

rootObj.build(rootNode) 

# Enable Python to collect the space used by the DOM. 

dec — None 

SWshe Sieeloble qvimeice (ee Pamll wetecshom— il 0! Sia! )) 

rootOb].export (sys.stdout, 0, name —"contact—list", 
namespacedef_='') 

rootObj.upper () 

Sys (So eCOlEewemeen( Ya = NGG) 

sys.stdout.write('\n') 

FOOLObDT export (Sys.stdout, 0, mamexs—"contacth—list", 
namespacedef_='') 

return rootObj 


USAGE_MSG = """\ 
Synopsis: 
Sample application using classes and subclasses generated by 
CSS 12 ches IDS iON 
Wisialcics: 
python member_specs_test.py infilename 


ww 


def usage(): 
print USAGE_MSG 
sys.exit (1) 


Gem amaaene()\e 


args = sys.argv[l1:] 

if len(args) != 1: 
usage () 

infilename = args[0] 


process (infilename) 


allie name == main Le: 


main () 


Notes: 


e We copy the function parse () from our generated code to serve as a model for 
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our function process (). 

e After parsing and displaying the XML instance document, we call method 
upper () in the generated class contact listTypeSub in order to walk the 
DOM tree and transform each xs: st ring to uppercase. 


4.7.1.4 Step 4 -- run the test application 


We can use the following command line to run our application: 


S python member_specs_test.py member_specs_data.xml 


When we run our application, here is the output: 


S python member_specs_test.py member_specs_data.xml 
<?xml version="1.0" ?> 
<contact-list locator="http://www.rexx.com/~dkuhlman"> 
<description>My list of contacts</description> 
<Contacte priority — 0. 050000" collor=code—"red! ad—"il'> 
<first-name>arlene</first-—name> 
<last-name>Allen</last-name> 
<interest>traveling</interest> 
<category>2</category> 
</contact> 
<<) @imicalere — I asic 


<contact-list locator="HTTP://WWW.REXX.COM/~DKUHLMAN"> 
<description>MY LIST OF CONTACTS</description> 
<COntace Pelomlty— Oe OoUOCO™ Wwolou[ecode— RE Di sad — Mite 
<first-name>ARLENE</first-—name> 
<last-name>ALLEN</last-name> 
<interest>TRAVELING</interest> 
<category>2</category> 
—/ Combace> 
</contact—list> 


Notes: 


e The output above shows both before- and after-version of exporting the parsed 
XML instance document. 


4.8 Some hints 


The following hints are offered for convenience. You can discover them for yourself 
rather easily by inspecting the generated code. 


4.8.1 Children defined with maxOccurs greater than 1 


If a child element is defined in the XML schema with maxOccurs="unbounded" or 
a value of maxOccurs greater than 1, then access to the child is through a list. 
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4.8.2 Children defined with simple numeric types 


If a child element is defined as a numeric type such as xs: integer, xs: float, or 
xs: double or as a simple type that is (ultimately) based on a numeric type, then the 
value is stored (in the Python object) as a Python data type (int, float, etc). 


4.8.3 The type of an element's character content 


But, when the element itself is defined as mixed="t rue" or the element a restriction of 
and has a simple (numeric) as a base, then the valueOf_ instance variable holds the 
character content and it is always a string, that is it is not converted. 


4.8.4 Constructors and their default values 


All parameters to the constructors of generated classes have default parameters. 
Therefore, you can create an "empty" instance of any element by calling the constructor 
with no parameters. 


For example, suppose we have the following XML schema: 


<?xml version="1.0"?> 
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> 


<xs:element name="plant-list" type="PlantList" /> 
OKs GCOmMOle<UyiOS imleiniS— Pears Iyer 
Gas SsiSciienes> 
<xs:element name="description" type="xs:string" /> 
<xs:element name="catagory" type="xs:integer" /> 
<xs:element name="fertilizer" type="FertilizerType" 
maxOccurs="unbounded" /> 
</xs:sequence> 
<xs:attribute name="identifier" type="xs:string" /> 
</xs:complexType> 


<xs:complexType name="FertilizerType"> 
G2 © SScilienies> 
<xs:element name="name" type="xs:string"/> 


<xs:element name="description" type="xs:string"/> 

</xs:sequence> 

<xs:iattribute name="id" type="xs:integer" /> 
</xs:complexType> 


</xs:schema> 


And, suppose we generate a module with the following command line: 


S ./generateDS.py -o garden_api.py garden.xsd 
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Then, for the element named Plant Type in the generated module named 
garden_api.py, you can create an instance as follows: 


>>> import garden_api 

>>> plant = garden_api.PlantType() 
>>> AMpPOGE Sys 

>>> Planer ex pore (sys siecdoute 0) 
<PlantType/> 
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